Add Existing Module

To add an existing module to the pipeline:

1. Click on the "Library" tab on the bottom right hand corner of the "LONI Pipeline v3.0" window
2. Click the desired module from the list and drag the icon into the open graph space to add to the pipeline.
3. When a module is added to the graph space, a window named "Edit Module: *name of program*" appears. Select the desired parameters to activate and click "OK".

Notes:

• Some parameters cannot be turned off as they are not optional and are required for the module to function correctly.
• The "Edit Module" window can also be opened by right-clicking on the module you want to edit and selecting "Edit Module..."
• Variables can also be bound to the parameters of a module using "Bind to new variable" and "Edit Bound Variable" buttons.

To add a new Parameter to a newly created module, complete the following steps

1. After you have created a name and path for you module and clicked "OK" a screen will pop up
2. Click on "New Parameter" at the bottom of the screen

1. EditExistingParameter

These are the steps to adding a new resource (a data variable) to the pipeline

• Select Workflow->Data
• Click on "New"
• Enter a name in the label field
• Enter additional information in the comment field
• Choose a type
• Add a value (see composing a value below)

Composing a value

The values of variables in the pipelines can be composed from multiple parts. These parts can be static (see Add a String) or dynamic (see reference, list url, list resource)

• In the "New Variable" window, click "Compose" or "Compose URL", depending on the type of value that you want to enter.
• "Compose" will bring up a new window that shows 4 buttons to compose various values:

Add a string

adds a string of text to the value.

Add a reference

adds a value from an existing pipeline variable to the new value being composed.

Add a list URL

reference a list file and input the values that are contained in the list file into the value. (Note: Each value in the list file must be on a separate line. The pipeline will then run through each of the values on the list.)

Add a List resource

similar to "Add List URL," except that it references a list file that already exists within the pipeline. (Note: See "Add List URL.")

• In composing a value, the order in which the values were inputted will be preserved, so make sure to check the "Current Text" box to see if the value is correctly formatted.

This page documents the steps to adding function definitions to the pipeline ontology

1. step 1...
2. step 2...

-- MichaelJPan - 03 May 2005

The following steps should be taken to add a new group or a new user in the UserAndGroupManagement dialog:

• Adding a new group
  1. Click the "New Group" button or right-click on the UserAndGroupManagement graph a select the "Add group..." menu option
  2. Enter a value for "Name"
  3. Enter a value for "Priority" (Default = 5)
  4. Enter a value for "Number of Processors" (Default = 1)
  5. Enter a value for "Access Level" (Default value = 1)
  6. EditGroupFilePermissions as necessary.
  7. Click on the "Save" button to save changes
  8. Click on the "Close" button to close the dialog box

• Adding a new user
  1. Click on the "New User" button or right-click on the UserAndGroupManagement graph and select the "Add user..." menu option
  2. Enter a value for "Name"
  3. Enter a value for "E-Mail Address"
  4. Enter the user's password in both the "Password" and "Confirm Password" fields. These must match.
  5. Click on the "Save" button to save changes
  6. Click on the "Close" button to close the dialog box

New groups and users are automatically added as children of the root group. To change this, you will need to explicitly SetParentForUserOrGroup.

-- MichaelJPan - 12 Sep 2005

Description:

This feature allows developers to add any third-party external jar file as an extension.

Instructions:

Creating an external jar file (yourExtension.jar):

1. Let us assume the working directory to be $WORKDING_DIRECTORY. This is the directory-level at which you will create your jar file.

2. Create a config directory:

mkdir $WORKING_DIRECTORY/config

3. Create a configuration file (configuration.xml):

${WORKINGDIRECTORY}/config/configuration.xml

4. The configuration.xml has the following XML format:

   
   <?xml version="1.0" encoding="UTF-8"?>

   <extensions>
      <extension>
   <runlevel>1</runlevel>
   <class>Helloworld</class>
   <function>Prints out String pointed to by Helloworld_Argument.class</function>
   <arguments>
          <argument>
             Helloworld_Argument
          </argument>
   </arguments>
      </extension>
   </extensions>   
    

5. In the XML:

   <class>: Denotes the class inside yourExtension.jar that will serve
            as an entry point. The class specified here needs to exist
            inside the Jar file and also define the 'main' method

   <Functionality>: Holds the comment which will be displayed against
                    this user extension

   <Argument>: Holds the class name that serves as an argument to the 
               Main class pointed to by <class>. Multiple arguments
               can be specified inside the <arguments> tab

6. Constructing your jar file:

   <$WORKING_DIRECTORY>: jar cvf yourExtension.jar *

This creates an external jar file preserving the directory structure. In essence, the Pipeline expects the configuration file to be present in:

yourExtension.jar!/config/configuration.xml

Loading a Jar file inside LONI Pipeline:

1. Run the pipeline

2. Click on Application --> Options

3. Click on Extensions Tab

4. Click on Add Extension

If the extension has been appropriately created you should see a line added which describes the extension you just added.

5. Click save and exit

Example

1. Use helloworld.jar as the extension

2. helloworld.jar contains the following files (apart from the class files and config dir):

Helloworld.java

Helloworld_Argument.java

StringToPrint^?.txt

3. Helloworld takes in an argument of type Helloworld_Argument. Helloworld_Argument reads the first

line of StringToPrint^?.txt and passes it to Helloworld, which then prints it out to System.out.

4. To change what gets printed, when helloworld.jar gets loaded, you can unzip the jar, change the

line inside StringToPrint^?.txt. Create the jar again and load it through the Pipeline, and the modified

string will be printed.

-- JagadeeswaranRajendiran - 16 Sep 2005

• helloworld.jar: helloworld.jar {Example extension}

A RemoteServerOptionsPanel represents the panel to be displayed in the ServerAdministrationPanelGUI when its corresponding button is selected in the ButtonBar.

This document outlines the steps necessary to add a new RemoteServerOptionsPanel in the remote server administration user interface.

1. Implement the RemoteServerOptionsPanel interface
   1. Should extend from AbstractGUI
   2. Implement interface methods
      1. getLabel(): Very short label (displayed in ButtonBar)
      2. getDescription(): A little bit longer label (displayed when selected)
      3. getIconFilename(): Path of icon image resource in "images/icons" (displayed in ButtonBar)
2. Register with ServerAdministrationPanelGUI
   1. Instantiated in addServerRemoteOptionsPanels() method B. Register with registerOptionsPanel() method

Note: Implementations exist for ontology (RemoteServerOntologyOptionsGUI) and user/group management (RemoteServerUserGroupOptionsGUI). The user/group management options panel also implements a listener interface (UserGroupRefreshListener). When the panel is registered with the ServerAdministrationPanelGUI, it is also registered with the RemoteServerManager to be notified of ServerResponse events that are pertinent to the UserGroupRefreshListener. This design was necessary due to Java's lack of a callback mechanism.

-- RonNor - 30 Jan 2006

This document outlines the steps necessary to add a new ServerCommand for remote server administration

1. Create the ServerCommand class
   1. Use package edu.ucla.loni.pipeline.server.command in the core package
   2. Extend from provided classes if possible
      1. AbstractServerCommand
         1. Basic implementation of interface methods
         2. get/set attribute methods
      2. RefreshServerCommand
         1. To request target data (TARGET_ONTOLOGY, TARGET_USERGROUPS, etc.)
      3. ReloadServerCommand
         1. To reload target data server side
      4. AddEditRemoveServerCommand
         1. To add/edit/remove target data
2. Add specific instantiation based on Element to ServerCommandFactory.getInstance() method
   1. Used by ServerCommandWrapper to instantiate the ServerCommand object
3. Add specific processing of ServerCommand in edu.ucla.loni.pipeline.server.ServerManager in server package
   1. processEvent() method

-- RonNor - 30 Jan 2006

ApplicationsManager is a class that manages the relationship between the external viewing applications and the LONI Pipeline.

-- JagadeeswaranRajendiran - 22 Feb 2005

Overview

ApplicationsManager is the class that manages the relationship between external visualization applications (like Slicer, Shiva and LOVE) and the LONI Pipeline.

Functionality

When a user selects an option to view a data file in a pipeline, an appropriate ViewerEvent is scheduled by the UI. The ApplicationsManager picks up the ViewerEvent and chooses an appropriate viewer. The class uses the mappings between datatypes to viewers according to predefined viewer configuration. If the datatype has been register, then, it searches for the enlisted viewer and (if found) launches the viewer with the data-input/ data output image as the argument.

File Dependencies

ViewerConfig.xml found in the config directory.

Event Dependencies

ViewerEvent - The event that any component needs to schedule to invoke the functionality of ApplicationsManager.

-- JagadeeswaranRajendiran - 22 Feb 2005

An ArgumentCell represents a PipeCell's argument in the PipeGraph.

Like PipeCells, ArgumentCells are a persistant object model. An ArgumentCellView represents a view of an ArgumentCell.

ArgumentCells support highlighting with the setHighlighted() and isHighlighted() methods.

It also holds size, location, and positioning information that is used for rendering.

-- RonNor - 01 Feb 2006

ArgumentCellView represents a view of an ArgumentCell.

An ArgumentCellRenderer is used to actually render the ArgumentCellView in the PipeGraph.

-- RonNor - 01 Feb 2006

Pipeline v.3.0 beta client has been releaseed. Please see StableBuild for more information

CCB Meetings

This is a list of the minutes of the CCB meetings related to the Pipeline

• Presentations
• SigFlowMeetings
• Miscellaneous
  • MeetingMinutes200510281400

ReleaseNumberConvention provides information as to our numbering scheme.

Upcoming changes in the next release

*

Current release

• 3.0 beta 6.3 released 28 february 2006

Previous releases

• 3.0 beta 6.2 released 15 february 2006
• 3.0 beta 6.1 released 3 february 2006
• 3.0 beta 6.0 released 25 january 2006
• 3.0 beta 5.4 released 18 january 2006
• 3.0 beta 5.3 released 10 january 2006
• 3.0 beta 5.2 released 20 december 2005
• 3.0 beta 5.1 released 22 november 2005
• 3.0 beta 5.0 released 10 november 2005
• 3.0 beta 4.0 released 18 october 2005
• 3.0 beta 3.2 released 4 october 2005
• 3.0 beta 3.1 released 13 september 2005
• 3.0 beta 3.0 released 31 august 2005
• client 3.0 beta 2.4 released 12 august 2005
• client 3.0 beta 2.2 released 5 august 2005
• client 3.0 beta 2.0 released 20 july 2005
• client 3.0 beta 1.2 released 12 july 2005
• client 3.0 beta 1.1 released 6 july 2005
• client 3.0 beta 1.0 released 24 june 2005
• client 3.0 beta 0.0 released 11 june 2005

Changing Profiles (without using GUI):

If the user has multiple profiles, the best way to reflect the changes directly:

1. Each profile has its own directory (with its own set of options)

Choose the profile that you want to change, cd into that directory

(eg

~/.pipeline/ConfigProfile_01)

Change options in appropriate configuration file (eg: executionConfig.xml)

2. ~/.pipeline/ directory always has the options of the profile currently in use (which is specified in configurationLoader.xml which just serves as a pointer)

3. Check configurationLoader.xml (for the profile being used currently)

Case 1: The profile that you changed is not the currently used profile

Action: You are all set -

Case 2: The profile that you changed is the currently used profile

Action: You would need to copy the changed option files (in this case: executionConfig.xml) into ~/.pipeline from (eg: ~/.pipeline/ConfigProfile_01)

--

Adding Configuration Options:

The following is the description of an option in serverConfiguration.xml -

--

For an option with a single value:

<option name="Hostname.Aliases" value="pipelinev3.loni.ucla.edu">

</option>


In this case, the option just has a name and value associated with it - So
to add new options with only a single value, one would just have an option
resource under 'options'
--

For an option with multiple choices:

  <option name="UserAccountManager" value="CURSUserAccountManager">

    <choices>

      <choice>

        <name>

          XMLUserAccountManager

        </name>

        <value>

          edu.ucla.loni.pipeline.security.XMLUserAccountManager

        </value>

      </choice>

      <choice>

        <name>

          CURSUserAccountManager

        </name>

        <value>

          edu.ucla.loni.pipeline.security.CURSUserAccountManager

        </value>

      </choice>

    </choices>

  </option>

The name {in eg: UserAccountManager} has two possible values - XMLUserAccountManager and CURSUserAccountManager - And each of those values
can have an actual value associated with them -

This level of indirection was needed to keep the names user-readable - To add
another value as a choice for UserAccountManager - one would just have to add
a choice resource with an appropriate name and value

The current value associated with UserAccountManager is an attribute of the
option -

--

-- JagadeeswaranRajendiran - 06 Jan 2006

At the top of every pipeline module file is a set of class definitions. These class definitions define the various classes of objects to be manipulated, e.g. File, Function, Pipeline. Definitions are very useful, as they can denote inheritance relations. For example, SystemExecutable is a subclass of both File and FunctionSingleton, and is written as

  <rdfs:Class rdf:ID="ExecutableFile">
    <rdfs:SubClassOf rdf:resource="File" />
    <rdfs:SubClassOf rdf:resource="FunctionSingleton" />
  </rdfs:Class>

Once the classes have been defined, class instances are defined. In OWL, all objects are resources, and provided an ID, to which it can be referred. To define an Analyze image file,

  <AnalyzeImageFile rdf:ID="AnalyzeImageFileInstance1" />

To configure the client, in the Pipeline menubar, click on "Application", then "Options...". A configuration dialog should appear allowing for configuration.

• ConfigureExtensions
• ConfigureExecution
• ConfigureGUI
• ConfigureViewers

• InstallingPipelineApplication
• RunningPipelineApplication
• SoftwareRequirements

• PipelineGlossary

• UpgradingFromV1

• LibraryModules
• PipelineModules

• ShortUsageTutorial

• ClientConfiguration
• ChangingConfigurations (how to change the configuration options without using GUI)
• HotKeyNavigation

• MultipleValuedArguments
• CreatePipelineLists

• RunningMatlabScripts
• RunningRScripts

• TroubleShooting
• ReportBugs

Documents

• Client manual: This document is a working draft of the Pipeline manual. Feedback welcome.

client 3.0.beta1.1 (released 6 July 2005)

• added archiving of files in postprocessing
• added pause and resume functionality for both local and remote execution
• corrected abort behavior
• optimized monitor behavior to wait until dependency have executed to begin
• conversion of v1 modules now correctly processes specified list files and ucf files
• Fixed bug that did not save prefix, suffix, or grouping properly

client 3.0.beta1.2 (released 12 July 2005)

• singleton status pane now displays validation errors for singletonmonitors
• fixed network communications due to various issues with digest key mapping
• miscellaneous interface fixes

client 3.0.beta1.0 (released 24 June 2005)

• possible connections highlighted (default off)
• added popup menu item for connecting parameters (default off)
• fixed bug with dragging connections between parameter cells
• added predicates
• group permissions now use predicates

client 3.0.beta2.4 (released 12 August 2005)

• note that this list includes improvements made to beta2.3
• Added Reset button to Run Panel
• Added a new Zoom panel below the GlobalViewGraph
• Improvements made to User/Group management
• Added a menu that shows the most recently opened/saved modules.
• Number of recently opened modules is configurable in the options dialog
• menu items (quit and about) now work with mac hotkeys
• digest algorithm now uses sha-512, defaults back to sha-1, can be dynamically specified
• workflow sync resources now transmitted both ways, from client to server as well as server to client

client 3.0.beta2.2 (released 5 August 2005)

• note that this list includes improvements made to beta2.1
• Fixed bug that would not allow a module to be moved if it was connected to a parameter of a module in a nested pipeline directly
• Submitting a module for execution now has 2 steps- submitting a load command, and submitting a execute command.
• EditModule dialog now shows values of the bound parameters as a tooltip when hovering over the bound resource
• Added button to EditModule dialog that allows editing of bound variables
• Module Info Pane has been added above the userlist/library/workinglist pane
• Fixed NPE in the GlobalViewGraph
• single-clicking on a module in the library now displays information about it in the Module Info Pane
• Added LONI Pipeline logo to splash screen and About dialog.
• By default, possible connections are not highlighted, animated, and no longer can be made by the popup menu option
• Created new splash screen framework
• Added a "Fit To Width" option for zooming
• Combined Contributers tab with main tab and displays information in a scrolling text panel
• Display LONI logo on canvas
• Added Run Panel which contains a Run button and a Stop button
• Moved Support information to the main tab in About dialog
• Fixed many issues related to the connections management gui, including password request

client 3.0.beta2.0 (released 20 July 2005)

• Fixed bug that allowed connections to be drawn by dragging on a parameter cell that wasn't an input or output parameter
• Added popup menu option to edit/view a parameter, dependingon whether or not it was user-defined
• Moved save/remove parameter buttons to the main button panel of the Edit Function Panel for easier access
• Changed popup menu item "New Parameter" so that it creates a new parameter in the edit function panel rather than using a seperate dialog box for a new parameter
• Removed New Parameter Dialog
• Moved view of variables to a new tree node called "Data" for each module in the User List/Working List
• Added ability to bind a variable by dragging/dropping data from tree to a parameter
• Added ability to copy/paste variables to the "Data" treenode in the Working List
• Added ability to drag a variable existing in one module to another module
• Combined user/group management into one dialog that uses a graph representation to show relationships
• Added ability to Add/Edit/Remove users and groups
• Added file permissions to user/group management
• Miscellaneous interface fixes

client 3.0.beta0.0 (released 11 June 2005)

• GUI split into its own package
• included licensing info for external libraries
• xml files split into xul and config directories
• ontology definitions for local LONI execution included
• LONI theme packaged
• defined file containers in the ontology
• log messages are now searchable
• added application configurator

The clustering modules functionality allows for a researcher to select a group of connected modules, and abstract them as a single node. This action clusters them together, and nests the resulting pipeline as a sub-pipeline in the current pipeline.

To cluster a group of modules

• select the modules
• right-click to bring up the context menu
• click "Group modules"
  

• This will generate a grouped pipeline with all of the modules selected inside
  

• If the group of modules selected by the researcher rely on a module not currently selected, the grouping option will be disabled.
  

To ungroup a pipeline.

• right-click the pipeline you want to ungroup to bring up the context menu
• click "Ungroup"
  
• All of the modules inside the grouped pipeline will now be placed in the current pipeline.

• Pipelines- Pipelines are modules that nest other modules. All contained modules
• BranchPipelines^?
• IterationPipelines^?

Application directory

This directory is the location that the application uses.

Defaults to the tmp directory of the root file system

• c:\tmp\pipeline (on Windows)
• /tmp/pipeline (on Unix)

This value can be changed as follows:

- Go to config directory ${user.home}/.pipeline/config

- Edit executionConfig.xml

Find an option with name "cache.directory" Change value of choice associated with Unix_Cache to point to a directory on /cxfs/

- If you are trying to run a server on cerebro-sn1, please make sure value associated with option "cache.directory" is "Unix_Cache"

- Eg: if your config option looks like this:

  <option name="cache.directory" value="Unix_Cache">
    <choices>
      <choice>
        <name>
          Windows_Cache
        </name>
        <value>
          C:\Documents And Settings        </value>  
      </choice>
      <choice>
         <name>
          Unix_Cache
         </name>
         <value>/tmp/loniuser</value>
      </choice>
    </choices>
  </option>

then, you would change value associated with choice Unix_Cache to a directory on /cxfs/ where you as a user have permissions to write into -

  <option name="cache.directory" value="Unix_Cache">
    <choices>
      <choice>
        <name>
          Windows_Cache
        </name>
        <value>
          C:\Documents And Settings        </value>  
      </choice>
      <choice>
         <name>
          Unix_Cache
         </name>
         <value>/cxfs/tmp/loniuser</value>
      </choice>
    </choices>
  </option>

- copy executionConfig.xml to ${user.home}/.pipeline/config/default to complete the process

Enable grid engine

Defaults to false. If configured to use the grid engine, then need to also specify a preferred grid engine monitor class. Currently defaults to DRMAAGridProcessMonitor^?

Please refer to:

http://www.loni.ucla.edu/twiki/bin/view/Pipeline/AddingExtensions

-- JagadeeswaranRajendiran - 21 Oct 2005

The following is an explanation of the options that are configurable that affect the user interface:

• Canvas grab and scroll factor
  • How fast the user can pan a Pipeline canvas while holding shift and dragging the mouse on the canvas. The lower the value, the shorter the distance panned.

• Number of recent modules
  • Number of modules that are tracked in the Workflow->Recent Modules menu item in the Pipeline menubar

• On clicking parameters, highlight possible connections
  • When a user clicks a parameter that is an input/output, parameters that could possibly be connected are highlighted. In large pipelines, enabling this option may slow down performance.

• On clicking parameters, animate possible connections
  • Same as above option, except the parameters blink. In large pipelines, enabling this option may slow down performance.

• Enable popup menu item for connecting to parameters
  • Right-clicking on an input/output parameter adds a "Connect to" menu to the popup menu. This menu shows a list of parameters that could possibly be connected with the selected parameter, categorized by their modules. In large pipelines, enabling this option may slow down performance.

-- MichaelJPan - 28 Sep 2005

Network port

This is the port on which the server is listening for connections

Defaults to 8000

The server can alias a particular hostname. This is particular helpful when multiple Pipeline servers lie behind a load balancer. To alias a host name ...

Pipeline server settings can be configured.

• ConfigureExtensions
• ConfigureExecution
• ConfigureNetworking
• ConfigureCacheDirectory

Generic Viewer

1. When one clicks on a parameter with type (eg: AnalyzeImageFile^?, extension 'img'), if a viewer does not already exist, the following screen will pop up:

2. The user has to fill up the following two fields:

   A] Executable {mandatory} - an executable file with full path 
   B] Description {optional} - text describing the viewer

3. Click OK and the viewer will be launched

4. The viewer configuration is available for immediate use {user does not have to restart application}

5. Press OK to save configuration settings

Using SHIVA

1. shiva.module is automatically packaged with the application

2. For any data-type which is compatible with SHIVA, and the user intends to associate more data-types with SHIVA, the following steps will help integration:

   A. Executable {mandatory} - enter the full path to shiva.jar
   B. Module {mandatory} - change to shiva.module
   C. DataResourceID {mandatory} - enter 'SHIVAimagefile'
   D. Description {optional} - enter text describing the viewer

3. Press OK to save configuration settings

-- JagadeeswaranRajendiran - 21 Oct 2005

The Pipeline need the user to explicitly specify that two different modules are connected. Two modules are automatically connected if one of each of their arguments bind to the same resource. That is, if one module has the same input as the output of another module, the Pipeline automatically knows that the two modules are connected.

There are two ways in which the GUI allows for two modules to be connected.

• An argument from each of the modules are bound to the same resource
• In the GUI, click on an argument, and drag the mouse to another argument that you would like to connect. The application will automatically ensure that they bind to the same resource. Note: the GUI will not allow you to connect two arguments if they have incompatible datatypes.

Note: multiple modules can be connected by all binding to the same resource.

an example of a successful connection between two modules

If you want to undo the connection, simply right click on either parameter the module is connected to (square icon on edge of module) and click "Unbind".

as a guest (default) you will be able to view all the pages, but cannot edit. to edit the pages, these are the steps to start making modifications on wiki

1. register yourself (ie get a wikiname) at TWikiRegistration
2. talk to MichaelWhang or AndrewLee about getting a password using your wikiname
3. have MichaelJPan add your wikiname to the PipelineUsersGroup

before continuing, test out wiki's features in the Sandbox

or try the tutorial

once you have completed the above, you can then modify any page/topic once you log in, by clicking on the "Edit" link at the bottom of every page. or if you want, you can create a new topic by searching for it (make sure it's a wiki word) in the textbox next to "Go" at the top of the page. If the topic exists, you will be presented a link to it, which you can edit once you traverse that link. Otherwise, you will be presented with the option of creating that topic.

-- MichaelJPan - 11 Feb 2004

The core library contains the interfaces and classes that support the modular architecture of the Pipeline application.

The interfaces are

• Component
• Event

The classes are

• AbstractComponent
• AbstractEvent
• EventListener

The core library is released to allow developers who wish to do so, to add additional functionality to the Pipeline application via self contained components.

You can get the library here

-- MichaelJPan - 03 Mar 2005

You can create a new module using the following steps

1. Select, in the menu, "Workspace->New"
2. You will then be prompted for either a Pipeline (which contains other modules) or a system executable (which is a single module that wraps around a system executable).
3. Enter a name to which you would like to assign this module. The module will display this label as its name.
4. Enter a comment to which you would like to assign this module. Comments are helpful hints that will show up as tooltips.

• If you selected "pipeline", see CreateNewPipelineModule.
• If you select "System executable", see CreateNewSingletonModule.

Once you have created a new pipeline, the main canvas should now show a blank graph. You can now add modules to this pipeline by following the directions in CreateNewModule, or dragging exising ones into this pipeline.

If you selected "System executable" in CreateNewModule, you will now be prompted for the location of the program.

• You will then be prompted for either a Pipeline (which contains other modules) or a system executable (which is a single module that wraps around a system executable).

• you will be prompted to specify the location of your executable.
• Select the button to the right of the URL label textbox to choose a path. The application will automatically compose the URL for you.

Once completed, you will be prompted to specify the parameters to this program.

• Defining the parameters
  • AddNewParameter
  • EditExistingParameter

Once you have finished defining the parameters to this program, you will be prompted to edit the module.

• link here

Creating Pipeline Lists

Pipeline lists allow you to specify values for multiple iterations of a module. A Pipeline list can either be specified from a file (referred to as Pipeline list file), or it can be stored as a variable within a module file.

Creating a Pipeline List File

Pipeline list files are basic text files. Each line in the file represents a unique iteration value. A pipeline list file can be viewed, edited, and created using your favorite text editor. To reference a Pipeline list file within a variable, use the "Add List URL" button when editing a variable/argument.

You can also create a variable that represents a Pipeline list file. Follow the procedure to create a new data variable and simply set the class to "Pipeline list file". From here, select the list file as you normally would when creating a file variable.

Creating a Pipeline List

A Pipeline list is a special variable that acts like an external list file, but is stored as a variable within a module file. To reference a Pipeline list variable (including Pipeline list file variables), click the "Add List Reference" button when editing a variable/argument.

To create a Pipeline list, follow the procedure to create a new data variable and set the class to "Pipeline list". A Pipeline list variable can only contain a single value tab. Each "String" in the Pipeline list variable represents a unique iteration value.

Pipeline also provides a mechanism for creating a Pipeline list variable for a "File" or "File"-derivative class variable. When selecting the file path using the file chooser, it is possible to select multiple files. If multiple files are selected, a Pipeline list variable is automatically created with unique iteration values generated for each selected file. The file variable will also contain a list reference to the generated Pipeline list file. This new variable is initially labelled "List file for: {LABEL OF ORIGINAL FILE VARIABLE}".

-- RonNor - 29 Nov 2005

-- JonMorra - 09 Jan 2006

PLEASE DO NOT CREATE ANY MORE JDIALOGS IN PIPELINE

The DialogManager is intended to be the only way JDialogs are made. The DialogManager class has many classes for creating JDialogs under different circumstances (refer to the javadoc of DialogManager). When creating an AbstractGUI with the DialogManager it is unnecessary to add the CONTAINER or the COMPONENT to the inputs map (DialogManager takes care of that for you). However, if you desire to have only one dialog open at a time (such as when editing a parameter) you must provide the proper string to hash the dialog by. Here is my hashing convention, if it is followed then every instance which can open this particular dialog will operate correctly. In the below examples Class is the name of the GUI class that is being created.

If the dialog is independent of the activeModule then use Class.class.getName() as the hash string.
If the dialog is dependent on the activeModule then use activeModule.toString() + Class.class.getName() as the hash string.
If the dialog is editing a function then use function.toString() + Class.class.getName() as the hash string.
If the dialog is editing a module then use module.toString() + Class.class.getName() as the hash string.
If the dialog is editing a argument then use argument.toString() + Class.class.getName() as the hash string.
If the dialog is editing a class then use class.toString() + Class.class.getName() as the hash string.
If the dialog is editing a variable then use variable.toString() + Class.class.getName() as the hash string.

No permission to read topic Pipeline.DMRequests - perhaps you need to log in?

No permission to read topic DemoIssuesApril2006 - perhaps you need to log in?

We have revamped the Pipeline framework to be extension based. That is, the functionalities of the application will be modularized, and packaged into their own extensions, which can optionally be loaded by the user as desired.

The extensions framework is based upon event processing. The application on startup bood loads the extension manager, which then in turn, loads all the extensions. The application has a set of predefined events which the extensions can post. Alternatively, each extension can post self defined events. These events are posted by each of the extensions to the extensions manager, which in turn passes them onto other extensions. Upon receiving posted events, each extension checks to see if it should process the event. If so, it will. Otherwise, the event is ignored by that extension.

Currently, these are the extensions planned

1. ExtensionManager^? - manages all extensions
2. PipelineManager^? - manages all pipelines
3. ExecutionManager^? - manages execution monitors
4. ProcessScheduler
5. LogManager^? - manages all loggers
6. UIManager - manages user interface object instances
7. a graphical user interface
8. a command line interface
9. a provenance engine
10. a validation engine
11. SecurityManager^? - manages security issues
12. NetworkManager^? - manages network connections
13. ResourceManager^? - manages resources needed by the pipelines
14. ApplicationsManager - manages applications for users
15. an AI engine
16. a data store

This design allows for extensions to be added, so long as they are aware of the edu.ucla.loni.EventListener and edu.ucla.loni.Event classes. The extensions do not need to know which other extensions are loaded (unless they intend to specifically process events posted by another extension), nor do they even need to be aware of the extensions manager.

Users: submit comments here

-- MichaelJPan - 20 Feb 2004

• DesignGoal

• SourceCodeRepository
• CodingStyle^?

• NetworkingAPI
• UnitTesting
• AddingExtensions (how to create an extension for the Pipeline)
• UsingSwiXML (how to use XML to design user interfaces)
• SigningForWebStart (how to sign the jar file for webstart)
• UsingHibernate (how to use the database agnostic Hibernate persistence engine)

• OntologyDocuments
• FormatModuleForPipelineOntology
• AddOntologyFunctions

• CreatingDialogs
• PassingDataInGUI

• Javadoc documentation of source
• Doxygen documentation of source

• RNorExitDocumentation

• ChangeLog

• SourceCodeStatistics

• TroubleShooting
• ReportBugs

Workflow language description

The draft can be found here.

Diffie-Hellman key exchange (alternatively key agreement or key negotiation) is a cryptographic protocol which allows two parties to agree on a secret key over an insecure communications channel. This key can then be used to encrypt subsequent communications using a symmetric key cipher.

The Diffie-Hellman algorithm specifies that the shared secret key

k = g^(ab) mod p

where a is one's own secret key and b is the other's public key.

You can find out more about the Diffie Hellman algorithm in the Wikipedia

-- MichaelJPan - 22 Mar 2005

The LONI Pipeline is released under the LONI License.

The LONI Pipeline client v.3.0.beta6.3 has been released as of 28 February 2006.

A copy can be downloaded from the LONI download site.

A list of changes to this version can be found at ChangeLog

Please file bugs and features request at our bug tracking system.

Common issues are addressed at the TroubleShooting page

1 Enter a name in for "label"

1. Add a comment
2. Select Type- either "Command Line Option" or "File"
3. If File is selected then select "Connection Type"- either "Input" or "Output"
4. Add a prefix, if given, in the form "-nameofprefix"

1. Scroll down using the bar on the right
2. If the parameter is optional, make sure that the "Optional" checkbox is checked.
3. Click Save (*Note if you do not click save and simply click "OK" at the bottom of the window, the parameter you just entered will not be saved.)

1. To continue adding parameters click "New Parameter" and follow the steps outlined above.

To add a parameter to an existing module

1. Right click on module
2. Click on "Edit Function"

1. Click on "New Parameter"
2. Enter a label and comment
3. select type of parameter
4. if needed, specify connection type in drag-down list

1. Specify if optional or not
2. Click "Save" and then click "OK"

When adding or editing a group, you have the option to specify GroupPermissions for the particular group that you are adding or editing. A group may have zero or more FilePermissions. Permissions that are set for a directory are automatically recursive. Any permissions specified for a directory are inherited by that directory's sub-directories. Explicitly setting a permission for a subdirectory will override permissions inheritted by parent directories.

Follow instructions to EditUserOrGroup to view the group editing dialog.

• Adding a new FilePermission
  1. Click on the "New File Permission" button
  2. A file selection dialog will open. Select the file/directory that you would like to specify permissions for
  3. The file/directory you selected will be added to the end of the table of FilePermissions
  4. Click the "Save" button to finalize changes to the group

• Editing an existing FilePermission
  1. Click on the checkbox for read, write, or execute permissions for FilePermission you are editing to toggle the respective permission value
  2. To enable a permission, check it's value
  3. To disable a permission, uncheck it's value
  4. Click the "Save" button to finalize changes to the group

• Removing an existing FilePermission
  1. Click on the FilePermission you wish to remove to highlight it
  2. Click the "Remove File Permission" button
  3. The FilePermission should no longer appear in the FilePermissions table
  4. Click the "Save" button to finalize changes to the group

-- MichaelJPan - 12 Sep 2005

The following steps should be taken to edit a group or user in the UserAndGroupManagement dialog:

• Editing a group
  1. Double-click the group you would like to edit. This is typically represented by a green node, with the exception of the root node, which is gray.
  2. Modify values for "Name", "Priority", "Number of Processors", and "Access Level" as necessary
  3. EditGroupFilePermissions as necessary
  4. Click on the "Reset" button to revert changes
  5. Click on the "Save" button to save changes
  6. Click on the "Close" button to close the dialog box

• Editing a user
  1. Double-click the user you would like to edit. This is represented by a blue node.
  2. Modify values for "Name", "E-Mail Address", and "Password" as necessary
  3. If modifying the "Password" field, the "Confirm Password" field must match.
  4. Click on the "Reset" button to revert changes
  5. Click on the "Save" button to save changes
  6. Click on the "Close" button to close the dialog box

-- MichaelJPan - 12 Sep 2005

The Pipeline networking library provides for secure communications using TCP. The encryption protocol uses DiffieHellman algorithm to allow for a client and server to exchange a secret key over an insecure medium without any prior secrets. The steps involved for extablishing a connection in the Pipeline are as follows

• NetworkingAPIEstablishTCPSocket
• NetworkingAPIEstablishDiffieHellmanParameters
• NetworkingAPIEstablishAsymmetricKeys
• NetworkingAPIEstablishRemotePublicKey
• NetworkingAPIEstablishEncryptionCiphers
• NetworkingAPIAuthenticateUser
• NetworkingAPIUserPermissions
• NetworkingAPIWorkflowsSynchronization
• NetworkingAPITransmitPendingPackets

Using the Execution Wizard

To open the Execution Wizard:

1. Go to "Tasks" menu
2. Click on "Execution Wizard..."
3. Follow the instructions in the window.

Functions of the Execution Wizard

• Can activate or deactivate modules within the module pipeline.
• Can activate or deactivate module parameters within the module pipeline.
• Can bind specific data (datasets/variables) to module parameters within the module pipeline.
• Create or remove connections between modules within the module pipeline.

Steps for the Execution Wizard

• Open desired module pipeline (Workflow -> Load)
• Click "Next" after introduction window.

• Select modules to be activated/deactivated by checking/unchecking boxes to the left of the module names. Click "Next."

• Select module parameters to be activated by clicking on the desired module, then clicking on box next to the desired parameter to activate/deactivate. (Note: Some parameters cannot be deactivated. These parameters are required for the module to function correctly.)

• Bind desired datasets/variables to module parameters by selecting the module parameter and clicking on the drop-down menu to select a different variable, or create a new variable (see AddNewResource) using the "New Variable" button. (Note: The Description pane for each parameter usually shows the default values that are assigned to the parameter, so refer to the description if you are unsure of the value binded.)

• Click "Finish" to apply changes.

LONI Pipeline makes extensive use of external libraries. In order to avoid potential licensing issues which may arise with the distribution of these packages, you will need to download these packages and ensure that they are located in your classpath.

JGraph is a robust and complete graph component. It is released under a mix of licenses, i.e. BSD, MPL, LGPL, and their own JGraph General License. The two packages from JGraph needed by the Pipeline are

• jgraph.jar
• jgraphaddons.jar

L2FProd provides dynamic Look&Feel customization for Java applications, as well as some common user interface components. It is released under a BSD-style license. The two packages from L2FProd needed by the Pipeline are

• skinlf.jar
• l2fprod-common-all.jar

SwiXML provides for user interface design in XML for Java applications. It is released under an Apache-style license. The package from Swixml needed by the Pipeline is

• swixml.jar

Note: SwiXML also requires JDOM (see next package).

JDOM is a Java representation of an XML document. It is released under an Apache-style license. The package from JDOM needed by the Pipeline is

• jdom.jar

Log4J is a logging service for Java applications. It is released under the Apache license. The package from JDOM needed by the Pipeline is

• log4j-1.2.8.jar

DRMAA binding is Sun's binding to the Sun Grid Engine. It is released under Sun's Sun Industry Standards Source License. The package needed by the Pipeline is

• drmaa.jar

JUnit is a framework for unit testing of Java development. It is released under the Common Public License. The package needed by the Pipeline is

• junit.jar

The [[http://www.sdsc.edu/srb/][Storage Resource Broker (SRB)] is UCSD's client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and accessing replicated data sets. The package needed by the Pipeline is

• jargon.jar

*jh.jar

-- MichaelJPan - 08 Feb 2005

• PipelineCollaborators
• FundingInstitutions

Other workflow systems

• Kepler
• SCIRun
• Taverna

File dependencies are another example of the usefulness of definitions. For example, Analyze image files have two separate parts- an image header file, and an image data file. These two have the same path, except with file extension "img" for the data file, and "hdr" for the header file. To describe this, we write

  <rdfs:Class rdf:ID="AnalyzeImageHeaderFile" extension="hdr">
    <rdfs:SubClassOf rdf:resource="File" />
    <rdfs:label><![CDATA[AnalyzeImageHeaderFile]]></rdfs:label>
    <rdfs:comment />
  </rdfs:Class>
  <rdfs:Class rdf:ID="AnalyzeImageDataFile" extension="img">
    <rdfs:SubClassOf rdf:resource="File" />
    <rdfs:label><![CDATA[AnalyzeImageDataFile]]></rdfs:label>
    <rdfs:comment />
  </rdfs:Class>
  <rdfs:Class rdf:ID="AnalyzeImageFile">
    <rdfs:SubClassOf rdf:resource="File" />
    <HasPart rdf:resource="AnalyzeImageHeaderFile">
      <replace type="extension"/>
    </HasPart>
    <HasPart rdf:resource="AnalyzeImageDataFile">
      <replace type="extension"/>
    </HasPart>
    <rdfs:label><![CDATA[AnalyzeImageFile]]></rdfs:label>
    <rdfs:comment />
  </rdfs:Class>

With this definition, an user can specify an AnalyzeImageFile, which the application will know that both files are included. Users can also specify AnalyzeImageDataFile or AnalyzeImageHeaderFile individually, which the application will know that it is just that file with the particular file extension.

File permissions in the Pipeline follow the same schema for file permissions in most operating systems. That is, each permission specifies a path, and has a read, write, and execute boolean flag.

This module will run the Firefox browser, with an optional url input to be set as the start page.

Some notes:

• The node describes the executable. It specifies the url of the executable, its parameters, and human readable labels and descriptions.

Files

• firefox.module:

To format a pipeline module for use in pipeline ontology:

1. Open up *.module in Emacs or other text editor

2. In text editor, save another version of module as *.owl

3. Insert the following code before the "ExecutableFile" node:

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" 
         xmlns:owl="http://www.w3.org/2002/07/owl#">
  <owl:Ontology>
    <owl:versionInfo><![CDATA[20050412]]></owl:versionInfo>
  </owl:Ontology>

4. Make sure that

</rdf:RDF>

is on the last line following the final

</ExecutableFile>

5. Save changes to file

Freesurfer tutorial module

This is the Pipeline module for the recon-all script in the Freesurfer tutorial.

Screenshots

Usage

Some caveats:

• This requires Freesurfer distribution after 12 September 2005
• This requires the latest Pipeline release
• This has only been tested on Mac OSX 10.4
• The data directory variable needs to be changed to reflect the location of your tutorial data
• The freesurfer modules need to be changed to reflect the location of your freesurfer location
• The nu_correct modules need to be changed to reflect the location of your nu_correct location
• mri_watershed may possibly output a lot of stuff to stdout and crash the Java virtual machine. To fix this, enable the ad-hoc quiet option (ie, >& /dev/null). However, this causes problems on Mac OSX 10.4 (and possibly other systems) because the JVM will think that it has aborted.

Files

FreesurferTutorial.module

A pipeline at its core is a sequence of Functions applied to a data set. Hence function definitions are crucial. Since functions are resources as well, we can define functions just as easily as we define resources. Each function definition defines its parameters.

• FunctionParameters^?
• FunctionSingleton - a FunctionSingleton is an atomic function. This essentially wraps around a single executable.
• CompositeFunctions^?
  • FunctionPipeline - a FunctionPipeline is composed (recursively) of multiple functions.
  • BranchedFunction^?
  • IteratedFunction^?

This is an example of the function definitions index file used by the pipeline to load all the function definitions.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:owl="http://www.w3.org/2002/07/owl#">
 
  <Function rdf:resource="function/AIR/Affine2Rigid.owl"/>
  <Function rdf:resource="function/AIR/AlignLinear.owl"/>
  <Function rdf:resource="function/AIR/Align_Warp.owl"/>
  <Function rdf:resource="function/AIR/Binarize.owl"/>
  <Function rdf:resource="function/AIR/BinaryMask.owl"/>
  <Function rdf:resource="function/AIR/BinaryMath.owl"/>
  <Function rdf:resource="function/AIR/Cd_Air.owl"/>
  <Function rdf:resource="function/AIR/Cd_Warp.owl"/>
  <Function rdf:resource="function/AIR/Combine_Warp.owl"/>
  <Function rdf:resource="function/AIR/CombineAIR.owl"/>
  <Function rdf:resource="function/AIR/Crop.owl"/>
  <Function rdf:resource="function/AIR/DefineCommon_Air.owl"/>
  <Function rdf:resource="function/AIR/Determinant_Mask.owl"/>
  <Function rdf:resource="function/AIR/FixHeader.owl"/>
  <Function rdf:resource="function/AIR/Flirt2Air.owl"/>
  <Function rdf:resource="function/AIR/Fuse.owl"/>
  <Function rdf:resource="function/AIR/Gsmooth.owl"/>
  <Function rdf:resource="function/AIR/Identify.owl"/>
  <Function rdf:resource="function/AIR/InvertAIR.owl"/>
  <Function rdf:resource="function/AIR/Layout.owl"/>
  <Function rdf:resource="function/AIR/Magnify.owl"/>
  <Function rdf:resource="function/AIR/MakeAHeader.owl"/>
  <Function rdf:resource="function/AIR/ManualReslice.owl"/>
  <Function rdf:resource="function/AIR/Mv_Air.owl"/>
  <Function rdf:resource="function/AIR/Mv_Warp.owl"/>
  <Function rdf:resource="function/AIR/Overlay_Mask.owl"/>
  <Function rdf:resource="function/AIR/Reconcile_Air.owl"/>
  <Function rdf:resource="function/AIR/Reorient.owl"/>
  <Function rdf:resource="function/AIR/Resize.owl"/>
  <Function rdf:resource="function/AIR/Reslice_UCF.owl"/>
  <Function rdf:resource="function/AIR/Reslice_Unwarp_UCF.owl"/>
  <Function rdf:resource="function/AIR/Reslice_Unwarp.owl"/>
  <Function rdf:resource="function/AIR/Reslice_Vector.owl"/>
  <Function rdf:resource="function/AIR/Reslice_Warp_UCF.owl"/>
  <Function rdf:resource="function/AIR/Reslice_Warp.owl"/>
  <Function rdf:resource="function/AIR/ResliceAIR.owl"/>
  <Function rdf:resource="function/AIR/Reunite.owl"/>
  <Function rdf:resource="function/AIR/Scan_Vector.owl"/>
  <Function rdf:resource="function/AIR/Scan_Warp.owl"/>
  <Function rdf:resource="function/AIR/ScanAir.owl"/>
  <Function rdf:resource="function/AIR/ScanHeader.owl"/>
  <Function rdf:resource="function/AIR/Separate.owl"/>
  <Function rdf:resource="function/AIR/SetHeaderMax.owl"/>
  <Function rdf:resource="function/AIR/SizeOf.owl"/>
  <Function rdf:resource="function/AIR/SoftMean.owl"/>
  <Function rdf:resource="function/AIR/SoftMeanNoRescale.owl"/>
  <Function rdf:resource="function/AIR/Zoomer.owl"/>

</rdf:RDF>

This FunctionPipeline refers to 6 different functions, but only 5 distinct ones. Additionally, even though it referes to CropExecutableFile twice, it remaps the functions' IDs to Crop0 and Crop1, so that they can be distinguished from each other.

  <FunctionPipeline rdf:ID="simplealignGrid">
    <rdfs:label><![CDATA[simple align]]></rdfs:label>
    <GUI>
      <Icon rdf:resource="simplealignIcon" />
    </GUI>
    <Assign rdf:ID="Crop0">
      <owl:allValuesFrom rdf:resource="CropExecutableFile" />
    </Assign>
    <Assign rdf:ID="Crop1">
      <owl:allValuesFrom rdf:resource="CropExecutableFile" />
    </Assign>
    <Assign rdf:ID="AlignLinear">
      <owl:allValuesFrom rdf:resource="AlignLinearExecutableFile" />
    </Assign>
    <Assign rdf:ID="ResliceAIR">
      <owl:allValuesFrom rdf:resource="ResliceAIRExecutableFile" />
    </Assign>
    <Assign rdf:ID="InvertAIR">
      <owl:allValuesFrom rdf:resource="InvertAIRExecutableFile" />
    </Assign>
    <Assign rdf:ID="CombineAIR">
      <owl:allValuesFrom rdf:resource="CombineAIRExecutableFile" />
    </Assign>
  </FunctionPipeline>

-- MichaelJPan - 23 Feb 2005

  <ExecutableFile rdf:ID="CropExecutableFile"
        url="pipeline://localhost//usr/local/AIR/bin/crop">
        <Parameter connectionType="input" rdf:ID="CropVolume">
          <rdfs:label><![CDATA[Crop Volume]]></rdfs:label>
          <rdfs:comment><![CDATA[the name of the file to be cropped]]></rdfs:comment>
        </Parameter>
        <Parameter rdf:ID="CropPadding">
          <rdfs:label><![CDATA[Crop Padding]]></rdfs:label>
          <rdfs:comment><![CDATA[the number of voxels of padding 
             to add back after cropping 
             (more won't be added than was cropped)]]></rdfs:comment>
        </Parameter>
        <Parameter connectionType="output" rdf:ID="CropCroppedVolume">
          <rdfs:label><![CDATA[Crop Cropped Volume]]></rdfs:label>
          <rdfs:comment><![CDATA[the name of the output file]]></rdfs:comment>
        </Parameter>
        <Parameter rdf:ID="CropCLIOption4">
          <rdfs:label><![CDATA[Crop overwrite?(y/n)]]></rdfs:label>
          <rdfs:comment><![CDATA['y' grants permission to overwrite output]]></rdfs:comment>
        </Parameter>
        <Parameter connectionType="output" rdf:ID="CropTransformation" optional="true">
          <rdfs:label><![CDATA[Crop Transformation]]></rdfs:label>
          <rdfs:comment><![CDATA[the name of a .air file that will be created 
              to describe the crop with input as the reslice file and 
              output as the standard file.]]></rdfs:comment>
        </Parameter>
    <rdfs:label><![CDATA[Crop executable]]></rdfs:label>
    <rdfs:comment><![CDATA[executable for Crop]]></rdfs:comment>
  </ExecutableFile>

-- MichaelJPan - 23 Feb 2005

The following have generously contributed to the funding of the Pipeline development

• National Center for Research Resources (NCRR)
• National Centers for Biomedical Computing (NCBC)
• National Institute of General Medical Sciences (NIGMS)
• National Institute of Mental Health (NIMH)

ModulesPanesHolder

• Tabs should be created from an actual object with access to modules, variables, functions, classes, etc.
• PipelineDocument class
  • PipelineOntologyDocument to represent items in ontology
• Create a PipelineDocumentTreeModel class
  • PipelineTreeNode
    • Contextul menus
    • Highlight color
    • Drag and drop
  • Root node, PipelineDocumentTreeNode
    • PipelineDocument as user object
  • Keep track of node collapsed/expanded states

PipeGraphUI

• Handler classes
  • Managing connections
  • Managing contextual menus

Workspace

• User-mode view vs. design-mode view of Pipeline
  • Design-mode uses PipeCanvas
  • User-mode should provide a simplified overview of root Pipeline only
    • Provide interface for argument inputs/outputs

Miscellaneous

• Improved edge-routing in the graph
• JGoodies Forms (under BSD license) for cleaner input dialog layout
• Provide a LONI DOT graph layout server on high-availability.
  • Will need to change URL in PipeGraph.GRAPH_LAYOUT_SERVER
  • May have to modify parsing of returned DOT graph in PipeGraph.formatFromDot()

-- RonNor - 02 Feb 2006

The GlobalViewGraph displays a miniature view of the main graph. The entire graph is scaled to fit into the area of the GlobalViewGraph for a quick overview of all the modules and connections contained within the graph.

Contained within the GlobalViewGraph is a viewbox (red rectangle) that represents the current viewable area in the main graph. The user can also control the view in the main graph by interacting with portions of the global view. By clicking on a point on the global view, the main graph will center it's viewport on the selected point.

Shift+Drag on the global view will move the viewbox around and synchronize the main graph viewport as well.

Dragging on the global view will draw a box that zooms the main view to match the rectangular area drawn. The dimensional ratio of the rectangle drawn is restricted to match the ratio of the main viewport.

-- RonNor - 31 Jan 2006

Group permissions in the Pipeline are resolved using logical connectives. These include (possibly nested) conjunctions, disjunctions, and negations of a FilePermission.

Permissions to access a file is resolved using the AND operator on the file and all its containers, all the way up to the root file system. To have read access to /root/level1/level2/file, read access must be permitted to

• /root,
• /root/level1/,
• /root/level1/level2, and
• /root/level1/level2/file

Furthermore, a group can only have permissions to a file if all its parents allows it. So if a group's place in the group tree is /root/parent1/parent2/self, then in order to have read access to a file, read access must be permitted to

• root group,
• parent1 group,
• parent2 group, and
• self

However, each group can possible have multiple parents. So it is possible that a group belongs on 2 paths of a group tree, eg

• /root/parent1/self, and
• /root/parent2/self

In this case, permissions are resolved using the OR operator. Thus a group only needs permissions through one of its paths. In our example, if parent1 disallows access to a file, but root, parent2, and self all allow it, the self has permissions.

Features

Directory: /nethome/users/{user-name}/{pipeline-dir}/pipeline/pipeline/
Filename: groups.xml

Format

<Groups>
<Group name="your-groupname" parent="your-parentgroupname" priority="yourpriority" numProcessors="your-numberOfProcessorsAllowedForGroup"/>
<Group name="neuro" parent="0" priority="10" numProcessors="20">
<Group name=...    .../>
<Group name=...    .../>
...
...
<Group name=...    .../>
</Groups>

Variable Definitions

groupname: : Name of the group
parentgroupname: : The name of the parent group that this group belongs to
priority: : The priority at which the processes executing in the context of this group should run
numProcessors: : The number of processors that this group is allowed

Rules

1. The root group is identified when the group definition provides a string "0" (the numeral Zero as a string) under parentgroupname
2. Each group A's definition should preceed its usage as a parent for any other group B. For eg., if B is A's child,
   the definition of A should preceed that of B
3. The words "grid", "modulemonitor", "systemhelperprocess" etc should not be used for group names as they are keywords
4. The higher the numeric value of priority, the lower its semantic priority! Just like Unix, 0, 1, 2 have higher
   priority than 3 than 4 than 5 etc.
5. On any given branch of the group tree (that results from this file), the numProcessors should monotonically increase from the child to the root. That is, if A is the root Group, and B is a child of A and C is a child of B, then,

         {numProcessors(C)  <= numProcessors(B) <= numProcessors(C)} 
   
   should be ensured for all branches of the tree!
   

-- JagadeeswaranRajendiran - 22 Mar 2005

These are the hot keys that help navigate through the Pipeline application

• Shift-mouse: Pan graph (both main and global)
• Alt-shift-mouse: Zoom (main graph only)

Other available hotkeys

• Ctro-drag node: copies node

How To Use the Wiki

We encourage you to change the contents of a Wiki web page and to add your own web page topics - that is what a Wiki is all about !

Edit a Wiki Page

• Locate the rectangular tab containing the words "Edit this Page"
• Click the button.
• Enter a Wiki user name and password if prompted to do so.
• Edit the page using the Wiki Text Formatting Rules
• Preview your changes to the wiki web page by clicking the rectangular button labelled "Preview Changes"
• Once you are satisfied with the web page, save the changes by clicking the "Save Changes" button.

Add a Wiki page

A Wiki web page is considered to describe a specific topic. To add a new topic:

• Edit an existing Wiki page and add a WikiWord defining the name of your new Wiki topic.
• Preview and Save your changes to the existing web page. Your new Wiki Topic will have a small question mark inserted after the word. This means the new topic has not yet been created.
• Click the new topic.
• Enter a Wiki user name and password if prompted to do so.
• You will be viewing a Wiki Edit web page. Enter the contents of your new topic.
• Preview the changes and revise until you are satisfied (see above for details). * Save the changes to your newly created Wiki Web page.

Adding Images to a Wiki page

*See "Attaching a file to a Wiki page", below.

Attaching a file to a Wiki page.

You may at time want to associate an external file with a Wiki topic. Examples of this might be to add the detailed agenda of a meeting, or a spreadsheet table containing experimental results,and so forth.

To do so,

• Ensure the file is on your local computer.
• Click the button labelled "Attach a file". You will be asked to locate the file you want to upload, and to supply a comment that describes the file.
• Click the "Upload File" button at the bottom of the upload page.

Your file will be sent to the Wiki web site and a table of file attachments to this web page will appear at the bottom of the web page if it not already there.

• If you select the "Link" checkbox before uploading the file and it is an image format recognized by our Wiki software (Twiki) the picture will appear in the web page. You may then Edit the page (see above) if you want to modify the text before the picture, or it's location on the web page.

Deleting an attached file

• Locate the attached file in table of files (usually at the bottom of the wiki web page).
• Click "Manage"
• Click "Move Attachment" (at the bottom of the management web page).
• Choose "Trash" from the drop-down Web menu of projects.
• Type "TrashAttachment" into the Topic text entry area.
• Click the "Move" button to delete the attachment.

You can find more details using file attachments

This package consists of the modules of the 22 ITK exercises that help with familiarizing the user to learning ITK.

Some notes:

• each of the functions in the modules need to have the attribute "url" updated to reflect the location of your compiled exercise

-- StephenTabaquin - 13 Apr 2005

ITK Multi-Modality rigid registration

This module is a Pipeline wrapper around MultiModalityRigidRegistration executable.

• MultiModalityRigidRegistration.module: Pipeline Module for ITK image registration using the MultiModalityRigidRegistration utility

Instructions can be found here

Files

Documentation of the same workflow in Kepler

This document describe the steps needed to extend the edu.ucla.loni.pipeline.net.URL class. This has to be different than java.net.URL (though have the same methods) because java.net.URL requires definition of a new URLStreamHandler^? (which in turn requires users to change their security policy file) and a URLConnection method (which could possibly be used, but we've already implemented pipe.net.Connection which supports encryption, etc).

This inheriting class requires

• a constructor URL(String s) which sets field url=s,
• a cache() method for downloading
• a sendCached() method for uploading

In the case of SRB, an subclass of this URL class will provide an interface to Jargon API. In the case of a local file, http, etc, it would act as interface to Sun's Java classes

-- MichaelJPan - 18 Apr 2005

ITK Installation

This pipeline module has been created for users who intend to automate the download and compilation process of the ITK software. To install ITK, please follow the instructions below, according to the version of LONI Pipeline that you have downloaded:

Version 1

• InstallITKModuleVersion1

Version 3

• InstallITKModuleVersion3

-- JagadeeswaranRajendiran - 22 Feb 2005

ITK Installation

This pipeline module has been created for users who intend to automate the download and compilation process of the ITK software. To install ITK, please follow the instructions below:

Windows Platform (under cygwin)

• Download the pipeline module Itkinstall_cygwin.pm
• This pipeline module is intended to be used with Cygwin on any Windows platform in conjunction with the Pipeline
• It is assumed that the user has installed cygwin on his/her Windows machine in "C:\cygwin\" directory
• Furthermore, the applications cmake, cvs should be already installed within cygwin as additional packages
• All commands, namely: cvs, cmake, mkdir etc: should be located under "C:\cygwin\bin\" directory

Instructions

• Download and install Java
• Download and install CMake
• Download the pipeline application
• Download the pipeline module Itkinstall_cygwin.pm
• Start cygwin by clicking on C:\cygwin\cgywin.bat
• On the cygwin prompt, change dir to the TARGET_DIR directory, e.g. "/usr/local/ITK/"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select File->Load Modules and load itkinstallpipeline.pm
• Click "OK" when the global variable tables (see below in "What to expect" for details on this) come up
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while

Unix Platform (most flavours):

• Download the pipelne module Itkinstall_unix.pm
• This can be used on unix platforms (including mac os x). however, only the bash shell has been verified
• This pipeline module assumes that the following commands are in your path- cvs, cmake, mkdir

Unix Usage:

• Download and install Java
• Download and install CMake
• Download the pipeline application (see link above)
• Download the pipeline module Itkinstall_unix.pm
• Create a target directory (TARGET_DIR) where you would like to install ITK, e.g. "~/namic"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select File->Load Modules and load itkinstallpipeline.pm
• Click "OK" when the global variable tables (see below in "What to expect" for details on this) come up
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while

What to Expect:

• An "Insight" directory (lets call it ITK_INSTALL_DIR) is created inside the TARGET_DIR - example: /usr/local/ITK/Insight/
• Pipeline will download Insight source and make files into ITK_INSTALL_DIR
• Pipeline will then create a directory named "binary" (ITK_BINARY_DIR) in ITK_INSTALL_DIR - example: /usr/local/ITK/Insight/binary/
• The Insight toolkit is built and all binary files are stored in ITK_BINARY_DIR
• ITK_BINARY_DIR is a self-sufficient directory with all the executables and can be transported as an ITK executable package

-- JagadeeswaranRajendiran - 22 Sep 2005

ITK Installation (Version 3)

This pipeline module can be used with Pipeline Version 3.

Unix Platform (most flavours):

• Download the pipelne module Itkinstall_unix.module
• This can be used on unix platforms (including mac os x). however, only the bash shell has been verified
• This pipeline module assumes that the following commands are in your path- cvs, cmake, mkdir

Unix Usage:

• Download and install Java
• Download and install CMake
• Download the pipeline application
• Download the pipeline module Itkinstall_unix.module
• Create a target directory (TARGET_DIR) where you would like to install ITK, e.g. "~/namic"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select Workflow->Load and load the module
• Right-click on each module, click on Edit Function
  • Change the paths of different executables (like cvs, cmake etc) to point to local executable-paths
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while smile

What to Expect:

• An "Insight" directory (lets call it ITK_INSTALL_DIR) is created inside the TARGET_DIR - example: /usr/local/ITK/Insight/
• Pipeline will download Insight source and make files into ITK_INSTALL_DIR
• Pipeline will then create a directory named "binary" (ITK_BINARY_DIR) in ITK_INSTALL_DIR - example: /usr/local/ITK/Insight/binary/
• The Insight toolkit is built and all binary files are stored in ITK_BINARY_DIR
• ITK_BINARY_DIR is a self-sufficient directory with all the executables and can be transported as an ITK executable package

-- JagadeeswaranRajendiran - 22 Sep 2005

• Itkinstall_unix.module: ITK Installer for Pipeline Version 3

+ ITK Installation

This pipeline module has been created for users who intend to automate the download and compilation process of the ITK software. To install ITK, please follow the instructions below:

Windows Platform (under cygwin)

• Download the pipeline module Itkinstall_cygwin.pm
• This pipeline module is intended to be used with Cygwin on any Windows platform in conjunction with the Pipeline
• It is assumed that the user has installed cygwin on his/her Windows machine in "C:\cygwin\" directory
• Furthermore, the applications cmake, cvs should be already installed within cygwin as additional packages
• All commands, namely: cvs, cmake, mkdir etc: should be located under "C:\cygwin\bin\" directory

Instructions

• Download and install Java
• Download and install CMake
• Download the pipeline application
• Download the pipeline module Itkinstall_cygwin.pm
• Start cygwin by clicking on C:\cygwin\cgywin.bat
• On the cygwin prompt, change dir to the TARGET_DIR directory, e.g. "/usr/local/ITK/"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select File->Load Modules and load itkinstallpipeline.pm
• Click "OK" when the global variable tables (see below in "What to expect" for details on this) come up
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while

Unix Platform (most flavours):

• Download the pipelne module Itkinstall_unix.pm
• This can be used on unix platforms (including mac os x). however, only the bash shell has been verified
• This pipeline module assumes that the following commands are in your path- cvs, cmake, mkdir

Unix Usage:

• Download and install Java
• Download and install CMake
• Download the pipeline application (see link above)
• Download the pipeline module Itkinstall_unix.pm
• Create a target directory (TARGET_DIR) where you would like to install ITK, e.g. "~/namic"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select File->Load Modules and load itkinstallpipeline.pm
• Click "OK" when the global variable tables (see below in "What to expect" for details on this) come up
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while

What to Expect:

• An "Insight" directory (lets call it ITK_INSTALL_DIR) is created inside the TARGET_DIR - example: /usr/local/ITK/Insight/
• Pipeline will download Insight source and make files into ITK_INSTALL_DIR
• Pipeline will then create a directory named "binary" (ITK_BINARY_DIR) in ITK_INSTALL_DIR - example: /usr/local/ITK/Insight/binary/
• The Insight toolkit is built and all binary files are stored in ITK_BINARY_DIR
• ITK_BINARY_DIR is a self-sufficient directory with all the executables and can be transported as an ITK executable package

-- JagadeeswaranRajendiran - 22 Feb 2005

-- JagadeeswaranRajendiran - 22 Sep 2005

VTK Installation

This pipeline module has been created for users who intend to automate the download and compilation process of the VTK software. To install VTK, please follow the instructions below:

Windows Platform (under cygwin)

• This pipeline module is intended to be used with Cygwin on any Windows platform in conjunction with the Pipeline
• It is assumed that the user has installed cygwin on his/her Windows machine in "C:\cygwin\" directory
• Furthermore, the applications cmake, cvs should be already installed within cygwin as additional packages
• All commands, namely: cvs, cmake, mkdir etc: should be located under "C:\cygwin\bin\" directory

Instructions

• Download and install Java
• Download and install CMake
• Download the pipeline application (see link above)
• Download the pipeline module Vtk_install_using_pipeline.pm
• Start cygwin by clicking on C:\cygwin\cgywin.bat
• On the cygwin prompt, change dir to the TARGET_DIR directory, e.g. "/usr/local/Your_VTK/"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select File->Load Modules and load Vtk_install_using_pipeline.pm
• Click "OK" when the global variable tables (see below in "What to expect" for details on this) come up
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while

Unix Platform (most flavours)

• This can be used on unix platforms (including mac os x). however, only the bash shell has been verified
• this pipeline module assumes that the following commands are in your path- cvs, cmake, mkdir

Instructions

• Download and install Java
• Download and install CMake
• Download the pipeline application (see link above)
• Download the pipeline module Vtk_Unix_Install_using_pipeline.pm: Vtk installation for unix
• Create a target directory (TARGET_DIR) where you would like to install VTK, e.g. "~/namic"
• Copy pipeline.jar into TARGET_DIR
• Run "java -jar pipeline.jar"
• Select File->Load Modules and load Vtk_install_using_pipeline.pm
• Click "OK" when the global variable tables (see below in "What to expect" for details on this) come up
• Select Execution->Validation to make sure everything is fine
• Select Execution->Run to run
• Read a book (both the cvs checkout and compilation takes a while

What to Expect

• An "VTK" directory (lets call it VTK_INSTALL_DIR) is created inside the TARGET_DIR - example: /usr/local/Your_VTK/VTK/
• Pipeline will download Insight source and make files into VTK_INSTALL_DIR
• Pipeline will then create a directory named "binary" (VTK_BINARY_DIR) in VTK_INSTALL_DIR - example: /usr/local/Your_VTK/VTK/binary/
• The VTK toolkit is built and all binary files are stored in VTK_BINARY_DIR
• VTK_BINARY_DIR is a self-sufficient directory with all the executables and can be transported as an VTK executable package

-- JagadeeswaranRajendiran - 22 Feb 2005

The Pipeline application is a self-contained, executable Java jar file. That means, as long as you have Java installed on your system, there is no additional installation process.

Note: Pipeline v3 client will not run on SGI IRIX machines, due to the fact that SGI has not updated their implementation of the Java Virtual Machine in years.

No permission to read topic InternalReports - perhaps you need to log in?

Hibernate has problems when running under JUnit. Fixes are proposed here. However, this does not seem to work for us yet.

Once you have it working, you can automate the testing using Ant. An example for such usage can be found here.

No permission to read topic JagadeeswaranCodeDocumentation - perhaps you need to log in?

The modules distributed with the Pipeline client are modules to programs that exist on the Pipeline server hosted at LONI. To run these modules, you will need a LONI Services account, as well as permissions to the Pipeline server.

The LogMessagesPane is a user interface that tracks log events. It is composed of two portions. The bottom portion is a panel that displays a table of log messages. The top portion is a panel that displays a table of log filters.

There are two different log tables and the naming might cause some confusion. A LogTable object is used to store log events in memory. It can be accessed from the LogManager. These classes are defined in the core, under the logging package. The update() method in LogMessagesPane dequeues log events (PipelineLogEvents) from the LogTable and adds them to a GUI table model, called the LogTableModel.

The second type of log table is the actual Swing table that displays the log messages. Its data is derived from the LogTableModel. When a user chooses to filter the log messages, it filters the data stored in the LogTableModel and does not directly access or modify the LogTable.

The table of log filters is based on an inner class called FilterTableModel. Existing LogFilters can be edited in an EditLogFilterPanel. It is possible to create a new LogFilter with a NewLogFilterPanel. A filter is composed of a set of Rules. When editing a LogFilter, all you're doing is editing the individual Rules that compose the LogFilter. The Rules are displayed in a table based on a RuleTableModel. The RuleEditor is used to edit the different table cells. The first column allows you to select which column in the LogTable the rule applies to. The second column allows you select the type of filter to apply. The third column in the rule table lets you specify the value to filter with for a rule.

When more than one rule is defined for a filter, the log event must match with all rules in the filter to pass through. The same logic applies for multiple filters in the FilterTableModel. However, only filters that are enabled are used to filter the log messages.

-- RonNor - 02 Feb 2006

The main menu manages all the menus and menu items that appear in the menu bar of the Pipeline GUI. It provides access to the RecentModuleManager and the execution menu items (run, pause, abort, and reset).

It also has a setActiveModule() method that disables certain menu items and menus if there isn't an active module, and enables them again if there is.

-- RonNor - 31 Jan 2006

The main panel is everything below the main menu bar. It defines many different panels and split panes that represent the main user interface window.

The top portion contains an execution panel, with buttons for starting, stopping, pausing, and aborting execution of a module.

Below that is a portion split into two parts vertically. On the left is another splitpane that contains the GlobalViewGraph, ZoomPanel, ModuleInfoPane, and the ModulesPanesHolder. On the right side is a scrollpane holding the Workspace.

The main panel is also responsible for updating the LogMessagesPane.

The setActiveModule() method is responsible for setting the workspace with the active module. It also updates the state of the run panel buttons to be consistant with the active module's execution status. If there isn't an active module, these buttons are disabled. While loading the workspace, a loading message is displayed in the canvas area that indicates the loading progress.

-- RonNor - 31 Jan 2006

• Teleconferences
  • TeleconferenceMinutes200508121000
  • NAMIC teleconferences
• External meetings
  • MeetingMinutesOctober2006PipelineSCIRun
  • MeetingMinutes200601111300
  • MeetingMinutes200508161030
• Internal meetings
  • MeetingMinutes200506071130
  • MeetingMinutes200501101000
  • MeetingMinutes200411121530
  • MeetingMinutes200409241400
  • MeetingMinutes200502171000
  • MeetingMinutes200502101300
  • MeetingMinutes200502101000
• CcbMeetings
• LONIR Meetings
• NAMIC meetings
• MBIRN

24 september 2004 2pm

present: ScottNeu, Karen Crawford, Rebecca Blanton, Amanda Hammond, Allan Mac Kenzie Graham, Craig Schwartz, MichaelJPan, and others

developed a list of features for future Pipeline versions

see PipelineFeatures

12 November 2004 3:30pm

present: ScottNeu, Allan Mac Kenzie Graham, MichaelJPan, Ivo Dinov, Arthur Toga, Jyl Boline, Daniel Valentino, Rico Magispoc

discussed status for Pipeline project

specified additional subcommittees- external project integration, visualization, documentation

discussed licensing issues

discussed search engine issues

10 January 2005 10am

present: MichaelJPan, Ron Nor, Jags Rajendiran

Ron and Jags brought up to date with Pipeline development and assigned various tasks.

Jags, as NAMIC liason, will work on developing pipelines that make use of the NAMIC toolkit, which includes ITK, VTK, and Slicer. He will also work on developing an applications manager and maturing the process scheduler.

Ron, having previously worked on the GUI, will continue work on it. He will begin with implementing a routing algorithm to draw pipeline nodes and their connections.

10 february 2004 10am

present: DavidRex, David Shattuck, ScottNeu, Karen Crawford, MichaelJPan

mapped out components of entire LONI Brain suite software, and component interactions

-- MichaelJPan - 11 Feb 2004

10 february 2004 1pm

present: DavidRex, Jeff Ma, MichaelJPan

mapped out components of Pipeline processing environment.

agreed to branch current trunk of Pipeline code, leaving main trunk for debugging the new release to users. new branch was created with the name "code-restructuring" to allow for major changes to fit the proposed extensions framework, and to add logging capabilities using this framework.

17 february 2004 10am

present: DavidRex, David Shattuck, ScottNeu, Karen Crawford, MichaelJPan

worked through a use-case, touched on the following things

1. need an automated tool selector extension which suggests best tools for a task based upon i/o requirements, data formats, applicability, etc
2. a goal oriented design for data processing
3. storage of provenance with file, throughout processing
4. error checking/quality assurance extension

7 june 2005 11:30am

present: Rico Magispoc, Karen Crawford, MichaelJPan

Pipeline database requirements

• User authentication
• Group management
• Runtime information
• Provenance
• Dynamic ontology distribution

In LONI, user authentication is provided for by CURS. The Pipeline to be distributed will allow external collaborators to use an XML based user authentication file.

Group management is not provided for by CURS. In order to allow for dynamic group management (e.g. scheduling policy, process & resource allocation, execution permission) we need a highly available database server.

Persistent runtime information will allow for recovery after a system failure. This does not need to be on high availability.

Provenance requires high availability.

Dynamic ontology distribution (planned, but far into the future) requires high availability.

Database manager

• PostgresSQL
  • Pros
    • Ensures data integrity regardless of access method
    • Feature support
      • Cursors (reduces bandwidth requirements)
      • Transactions
      • Foreign keys
      • Views
  • Cons
    • Not currently on high availablilty
• MySQL
  • Pros
    • Already on high availability
    • LONI administrators are familiar with it
  • Cons
    • GNU Public License (GPL)- this will be resolved as we intend to purchase a commercial license (need to verify the correctness of this assumption)
    • Feature support (see PostgreSQL pros). Most of these will be implemented. Timeline for completion by the organization is unknown

Pipeline and load balancing

For the LONI grid computing environment, we are looking at having multiple submit hosts, each of which hosts a Pipeline server. Access to a Pipeline server by a Pipeline client will be routed by a load balancer to ensure that no one machine is innundated with jobs.

Issues

• Generic load balancer is round robin, so cannot balance according to process.
• Generic load balancer does not have knowledge of a user's existing pipelines on a particular server
• Requires that runtime information not be located with each Pipeline server. Instead, persistent runtime information will need to exist on a highly available DBMS.
• Currently the Pipeline's authentication is tied to its execution, in that the same instance which authenticates does the execution. So its not really possible to load balance and have a user switch machines mid execution.

Conclusions and action items (in bold)

Database Manager

• In the short term, the Pipeline will use the LONI HA MySQL database
• Rico will purchase MySQL commercial license
• Karen will provide Michael with MySQL contact, and a development database for the Pipeline
• Michael will verify that the commercial license will not cause our code to be GPL'd
• Michael will design any Pipeline data schemas assuming a MySQL^? database
• Michael will implement database functionality in the Pipeline making no assumptions on DBMS

Load Balancer

• The Pipeline server will need to be split into two separate components.
  • One component will perform user authentication.
    • Once authenticated, it will determine the best available executor for the user.
    • If the user has a running pipeline on some executor, the user will be connected to it. Otherwise, the user will be connected to the least loaded server.
    • This method does not check for the resources that a user intends to use
    • A more intelligent design will be reviewed in the near future.
  • The second component will execute the Pipelines and perform all the data and process management tasks
• Michael will come up with a design for the (intelligent) PipelineLoadBalancer^?

Pipeline server beta release

• Given these additional requirements, the release of the Pipeline server beta will be delayed. Current timeframe is unknown, but estimated to be September 2005.

16 august 2005, 10:30am

present: IvoDinov, MichaelJPan, David Weinstein, Michael Callahan, McKay Davis, J. Davison de St. Germain, Jagadeeswaran Rajendiran, Daniel Valentino

Meeting highlights

• We saw the Pipeline and SCIRun updates.
• We discussed the technical design/implementation details of the two environment's schemas for module and network/pipelet representation (.module and .tcl).
• We learned about the 5 basic data-types utilized by SCIRun (matrix, color-map, nrrd, field and SceneGraph).
• Pipeline - SCIRun interoperability issues:
  • (Integration) Each environment can read/parse/write/run networks/pipelets (.net/.module) from the other.
  • (Communication) Each environment can seamlessly run modules from the other environment, as part of a larger internal flow network.

Plan

• We'll attempt to integrate SCIRun modules within the Grid Pipeline, first.
• Then, we'll go into making Pipeline modules executable from within SCIRun.
• We'll write converters for mapping between Pipeline and SCIRun files
• Pipeline will run SCIRun in batch mode first - later we'll talk about SCIRun server(s).
• The SCIRun team will help with: providing a functional SCIRun version with decoupled GUI/Compute components, and writing Pipeline XML descriptions from SCIRun .net files.

Action items

• LONI will download the dev version of SCIRun from Subversion repository to be used as the testbed.
• SCI will write a program to convert SCIRun XML files to Pipeline files to be used to create a prototype pipeline that can be submitted to SCIRun
• LONI will extend the Pipeline to write SCIRun .net files and spawn a SCIRun process

Agenda

• Introduction of members
• Identification of grand challenges
  • Application integration
    • Matlab
    • R
    • SCIRun
    • Slicer
    • XGrid
    • Shiva
    • Condor
    • SRB
    • IDA
  • Workflow description language goals
    • Parameter connection restrictions
    • Workflow usage restrictions
    • Bibilographic references
    • Iteration and branch monitors
  • Provenance
    • Reproducibility
    • Verifiability
  • Process selection and validation
    • Contextual constraints
    • Constraint relaxation
    • Constraint identification
  • Research publication integration
    • BibTex output
    • Textual data mining
  • Mobile computing
    • Fundamental Challenges in Mobile Computing
    • Mobile information access
• Formulate timeline

Minutes

In attendance: MichaelJPan, Daniel Valentino, Ivo Dinov, Tony Chan (via proxy), David Shattuck, Allan Mackenzie Graham, Eric Kan, Amanda Hammond, Cornelius Hojatkashani, Ron Nor, Jagadeeswaran Rajendiran

• Introduction of members
• Mike gave an introduction and identified main challenges
  • Application integration
    • Pipeline can run external app jobs without spawning a new job
    • IDL
    • Standard interface
    • Enabling Shiva to call Pipeline
      • clarification on batch mode pipeline job submission
  • Workflow description language goals
    • NAMIC - new control flow paradigm
    • Depends on what is available in current implementation
    • Adding new functionality does not break current data flow paradigm
    • Biblio ref
      • citing algorithms
  • Provenance
    • Reproducing history of Pipeline
    • Making it practical -> archival
  • Process selection and validation
    • Given a process to perform, how to determine if correct in context of running it
    • Given a description of what user wants to do, how to determine which programs are useful in that context
    • Expert to specify these dependencies
    • Implicit file conversion from one tool to another
    • Research publication integration
    • Mining publications to determine algorithms for studies
  • Mobile computing
    • Disconnected operation
    • File access transparency
    • User mobility with respect to dataflow processing
    • Process migration
  • Visual verification
    • viewers
    • pause/resume
  • Database ACID properties/compliance
    • parallel computing issues
• Mike proposed that integration of applications such as Matlab and/or R be placed at the top of the list
• The formulation of timeline of grand challenges identified above, was tabled until the following month to focus on a plan for dissemination, usability, and training of the Pipeline in the next month.
  • Usability
    • Training for using the Pipeline
  • Running jobs on the grid for users

Agenda

The agenda of this meeting is to address the the necessary computational resources necessary to improve labwide productivity.

• Goals
  • Interoperability
  • Accessibility
  • Efficiency
• Services and requirements
  • Wiki
    • View restricted pages
  • Calendar
    • sharable
    • public vs. private
  • Source code repository
    • External access
    • CURS integration
  • Issue management
    • Progress tracking
    • User feedback
  • Software distribution
    • Multiple file distribution
    • Group control
  • Mobile computing
    • Hardware requirements
      • More laptops (currently on sale at UCLA store)
    • Software requirements
      • Display
      • Matlab
    • Portable storage devices
      • Fast data transfer
      • License key for software such as Matlab
    • Checkouts
  • File systems
    • Mobile computing (See CODA)
    • Fault tolerance and stability
    • User transparency
  • Software development
    • Project management portal (eg GForge)
    • Automated build systems
    • Automated testing

Minutes

In attendance: David Shattuck, Scott Neu, Cornelius, Linda, Yuan, Ivo, Neal, Rico, Amanda, Karen, Michael

• Wiki
  • CURS integration is uncertain at this point
  • added a Project for the systems so that users know how to use the environment
• Calendars
  • Web-based calendar can be enabled for groups and individual users
  • Contact Linda for details
• CVS
  • machine is availabe, there is a persistent bug yet to be resolved
  • CURS integration is ready to be deployed
  • Timeline: 7 November 2005
• GForge
  • Ivo brought up the idea that we should try GForge, as determined by the CCB Principal's meetings in April 2005. most was under the impression that it was experimental.
  • offers a one stop portal for services
  • question is what GForge offers us that we current do not have available
  • additionally, for groups invested in the current infrastructure, what is the cost of migration?
  • install in testing environment. discussion postponed until we all have had a chance to evaluate
• Issue tracking
  • Establish projects for system, hardware, database, web, software, purchases
• Software distribution
  • multiple files distribution- 1 to 2 weeks (Linda)
  • most software issues related have been resolved
  • group managed file distribution
• Mobile computing
  • we all agree that laptops are needed
  • will bring number of laptops to 6 Macs and 6 Windows machines
  • Shared network drives available via samba now
• Grid software
  • R, Shape Viewer, AIR, Minc, Brainsuite, FSL, Paul's code, Data Explorer
  • Some of this has been compiled for Solaris
  • number additional licenses needed to be determined from syslogs and users directly
  • Timeline for others: 1 month
• Neal brings up an issue of having groups being informed of the infrastructure and usage restrictions and requirements
  • Local networks for individual groups
• File systems
  • Have 4 processor NFS export, will get more NFS servers to support various groups
  • Grid now isolated from the users
  • Users have to migrate data between /loni and /cxfs
  • One possibility- Pipeline copies file to staging area. brings up issues of permissions, because pipeline user may not have all permissions
  • Have containers for groups (separate mounts, etc) that migrates data automatically, and the groups access is slowed based upon their usage
    • this is probably the best solution
    • implementation to be determined and revisited in one week
• Software development
  • Mac testing hardware is sorely needed
    • issue to be tabled
  • allow root access to group leaders?
    • issues of application/operating system upgrades to be addressed

Agenda

• Roll call
• Status update
• Proposal of new features
  • integration with Pygr
  • integration with databases, eg GraphDB and BLASTgres
• Formulate of roadmap (1, 3, and 6 months)

Minutes

In attendance: Allan MacKenzie^?-Graham, Jagadeeswaran Rajendiran, Raneath Nor, Michael Pan

• differenting modules and functions and pipelines in the library/working list
• exporting of parameters
• retroactive interference
• windows windows issues
• list processing
• distributed database either trivial or very hard

Agenda

• Roll call
• Status update
• Deployment roadmap (2, 4, 6 weeks)
• Development roadmap (1, 3, and 6 months)
• Research roadmap (6, 12, 18 months)

Minutes

In attendance:

• MichaelJPan, Roger Woods, Ron Nor, Jagadeeswaran Rajendiran, Jon Morra

Deployment

• individual servers available
• minc programs and descriptors
• email users on completion

Development

• control flow constructs
• validation framework
  • pre-execution validation
  • view error source
  • view at runtime the commands being run
• enable drmaa process monitors to bring over message
• fault tolerance
• load balancing

Research

• parallelization strategies
• distributed computing
• provenance
• validation
• Converences
  • BioGrid06
  • SSDBM 2006
  • HPDGrid 2006
  • HPDC15
  • WORKS06
  • VLDB2006

SciRun Meeting @ SCI Institute

Present: Martin Cole, MichaelJPan, Jagadeeswaran Rajendiran

Minutes

• ...
• ...

Action items

• possible new xml format for .net files (SCI)
• application specific .net for use in client side loni (SCI)
• make Loni Package in SCIRun, populated with client side Loni batch mode calls. (SCI)
• batch mode SCIRun for use on server side Loni (SCI, future)
• calling SCIRun from Loni, write .net file with filename substitution, on provided .net (Loni)
• Discover if SCIRun can run server side (X server available?)
• Build a new .net from Loni interactively. (Loni)

Miscellaneous

• Possibly using a simple intermediate language (such as JSON) to provide usage information (see Slicer3 intergration)

Pipeline Meetings - Pipeline/SCIRun Interaction Meeting (TeleConference^? CAll, DATE!)

Pipeline Meetings - Pipeline/SCIRun Interaction Meeting (Tele-Conference Call, DATE!)

Participants:

• (Pipeline) ArashPayan, JagadeeswaranRajendiran, JiaweiTam, StottParker, RicoMagsipoc, CorneliusHojatkashani, IvoDinov
• (SCI) DavidBrayford, dav@sci.utah.edu, Martin Cole, David Weinstein

Aims of this joint (LONI/SCI) CCB Project

• To develop a plan for Pipeline and SCIRun integration and tool integration by April 2007.
• 2004 Plans: In the CCB/NCBC Core 2, we had an aim on Data and Algorithm Interaction. This broad aim had two sub-aims that pertained to Pipeline/SCIRun interaction (more details are included in the appendix at the end):
  • We will extend the Pipeline to incorporate some SCIRun tools
  • We will extend the SCIRun Environment to allow running the Pipeline workflows.

Action Items

• The minutes and action plan from this TCon are available here

Current Project State

• Both Pipeline and SCIRun have made great leaps forward in terms of extending their scopes, processing and visualization tool libraries and their validation using variety of imaging studies.
• With the exception of brief interaction efforts (Summer 2005, Pipeline/SCIRun Hands-on Meeting, and Summer 2006, CCB AHM), there has been little joint effort to make headway on this interaction between the two infrastructures.
• Pipeline data-modeling language is XML-based and solves the challenges identified by the pipeline user community. Similarly, SCIRun’s perl-based scripting language addresses the needs of the SCIRun users/developers. Both allow batch-mode execution. It’s not clear what the Venn-diagram of these 2 feature-sets looks like (how much commonality is there, what is the extend of the disjoint parts, what it may take to integrate these schemas or is it feasible to write a converter between them).
• Does it make sense to start by externally integrating PowerApps^? from SCIRun as Tools/Libs within the Pipeline, and vise-versa?
• SCI Segmentation PowerApp^? SCI NCMIR Project
  • Videos (Video 1 & Video 2)

Action Plan

• TCon early in the week of Oct 09-13, 2006. Suggested days/times:
  • Wed 10/11/06, 12 PM (PDT)
  • Wed 10/11/06, 2 PM (PDT)
  • Thu 10/12/06, 11 AM (PDT)
  • David Weinstein (SCIRun) and Arash Payan (LONI) will take charge, suggest TCon times and communicate the aims of this effort to their groups.
• Develop a plan with specific deliverables, milestones and a timeline for the next 6 months – in order to facilitate the progress reporting on this project for the April 2007 (3rd year) CCB progress report.
• Wiki page for this effort is at: http://www.loni.ucla.edu/twiki/bin/view/Pipeline/MeetingMinutesOctober2006PipelineSCIRun (SCI members that have not already registered to edit our Wiki resources, may consider doing so. You can read the Wiki pages, but not edit, unless you registered first).

Pipeline/SCIRun Interaction Specification:

• Summary: Visual programming environments allow users to view the stages of algorithms. By extending the Pipeline to incorporate SCIRun, we will increase the level of granularity at which the user can control a data process. This will also increase the availability of visualization methods within the Pipeline Environment. We will develop intelligent algorithms to automatically launch viewers that are appropriate for a given data type. Similarly, the Pipeline Environment will be available within SCIRun, allowing users access to a broad collection of algorithms. Pipeline processing protocols will be introduced that wrap our current perl and script language commands for gene splicing, cross-species homology computation and sequence alignment into graphical Pipeline and SCIRun modules for automated and robust genetic modeling.
• We will expand the capabilities of the Pipeline Processing Environment so that its users have access to SCIRun modules within the Pipeline environment. This will leverage SCIRun’s visualization abilities with the Pipeline’s integrative stance and massive processing architecture, thereby allowing for complex visualization integrated with the Pipeline. This will allow for client/server processing of SCIRun modules, with client side execution of visualization processing. Modules from SCIRun will integrate seamlessly in the Pipeline workspace receiving processed data and launching visualization and other user interactive windows, Figure 1.

Figure1: This figure illustrates the similarity and the interaction of the LONI Pipeline and the SCIRun environments.

One of the first benefits that this integration will produce is to provide low level processing routines in the Pipeline. Currently, to perform sequences of mathematical tasks, users must either write their own programs or make use of multiple executables within the Pipeline. The Pipeline/SCIRun integration will allow users to utilize the math, geometry, and field libraries of SCIRun to allow efficient low level processing from within the Pipeline. Pipeline will no longer require separate program launches for small tasks, instead it will pass these smaller manipulations to the SCIRun executer. External algorithms and executables will still be integrated with the Pipeline through descriptors without an API.

We will integrate the SCIRun interface into the Pipeline, which will provide access to SCIRun modules whenever SCIRun is detected on the host machine. The SCIRun modules will appear in the tools panel of the environment. Users will be able to drag and drop SCIRun modules into the Pipeline workspace and work with them as they would any other module. SCIRun modules will maintain a look and feel similar to their native SCIRun look and feel. This will allow the user to edit the method as it would be edited within the SCIRun environment.

Currently, the Pipeline and SCIRun both use XML to describe their modules. We will extend the Pipeline to understand the SCIRun XML by integrating these descriptors into the Pipeline XML schema. We will then use the SCIRun XML descriptions of components to build the Pipeline modules. We will provide mechanisms for data translation using the data mediation architecture, described in Section E.4. The LONI Debabeler Engine will translate data in the Pipeline into data types and formats that are understood by SCIRun.

The CCB Pipeline Processing Environment will start a SCIRun executor when a pipeline requires this functionality. The CCB Pipeline will make calls to the executer for specific functions when the dataflow of a Pipeline requires it. The SCIRun executer will provide a multithreaded approach, thus maintaining the parallelism that is dictated by the Pipeline.

Pipelines in SCIRun: SCIRun will be extended to interpret the LONI Pipeline XML descriptors, utilizing them to insert modules and pipelines into the SCIRun environment. SCIRun will make calls to a Pipeline engine, executing specified algorithms and pipelets. Again, the LONI Debabeler Engine will provide the necessary conversion of files and data to be exchanged between SCIRun and the Pipeline.

Figure 2: SCIRun2 with different component models: integrating raw data, image processing and high dimensional data visualization.

The knowledge management tools described in Section D.3 provide a mechanism to determine the nature of the data in a given file. This information will be used within the LPPE to automatically select visualization methods that are appropriate for data at any stage of a pipeline. When requested by the user, the LPPE will launch an appropriate viewer from the available SCIRun modules. We will also use these mechanisms to connect the SCIRun environment to data stored within the database environment (Section D.3). SCIRun will select appropriate visualization strategies to provide interactive viewing of stored raw and processed data. We will extend SCIRun to maintain the provenance of data processed within its environment. The CCB provenance tools will be extended to appropriately interpret the SCIRun results.

• Integration of the CCB Pipeline into the SCIRun/BioPSE Problem-solving Environment using the Meta Component Model

The SCI Institute has recently created SCIRun2, which is an update to the SCIRun/BioPSE system specifically built to allow the seamless operation of external applications within SCIRun. SCIRun2 provides a meta component model that allows components to be used together even if they do not share the same underlying component architecture. Currently, SCIRun2 provides support for the DOE Common Component Architecture, a model that is more familiar to programmers who have used the Object Management Group’s (OMG) CORBA (OMG June 1995) or Microsoft’s COM. While SCIRun2 will still support for older SCIRun dataflow components, we are planning support for CORBA, COM, Itk and Vtk and we will extend SCIRun2 to support the CCB Pipeline. As a result, SCIRun2 will allow the native utilization of SCIRun components, CCA components, CCB components and networks, Itk, Vtk, and others in the same simulation. This capability will provide a variety of components available to the application developer, without requiring buy-in to any one particular methodology.

Figure 2 demonstrates with a simple example how SCIRun2 handles different component models. Two CCA components, Driver and Integrator, and one CORBA component, Function, are created in the SCIRun2 framework. In this simple example, the Driver is connected to both the Function and Integrator. Inside SCIRun2, two frameworks are hidden: the CCA framework and the CORBA Object Request Broker (ORB). The CCA framework creates the CCA components, Driver and Integrator. The CORBA framework creates the CORBA component, Function. The two CCA components can be connected in a straightforward manner through the CCA component model. However, the components Driver and Function cannot be connected directly, because neither CCA nor CORBA allow a connection from a component of a different model. Instead, a bridge component is created. Bridges belong to a special internal component model that is used to build a connection between components of different component models. In this example, Bridge has two ports: one CCA port and one CORBA port. In this way it can be connected to both CCA component and CORBA component. The CORBA invocation is converted to request to the CCA port inside the bridge component. In the case of the CCB pipeline bridge components will be automatically generated. Thus, the meta component model will provide a plug-in architecture for the CCB pipeline component models. The abstracted components will be manipulated and managed by the SCIRun2 framework, while the concrete CCB pipeline components will perform the actual work. This will allow the CCB components to communicate among each other in the most efficient manner, while facilitating their connection to SCIRun visualization and other SCIRun2 components.

• Acknowledgements:
  • "This work was made possible in part by software from the NIH/NCRR Center for Integrative Biomedical Computing, P41-RR12553-07."
  • This work was funded in part by the National Institutes of Health through the NIH Roadmap for Medical Research, Grant U54 RR021813 entitled Center for Computational Biology (CCB). Information on the National Centers for Biomedical Computing can be obtained fromhttp://nihroadmap.nih.gov/bioinformatics.

Pipeline Meetings - Pipeline/SCIRun Interaction Meeting 10/12/06 - Minutes

Participants:

• (Pipeline) ArashPayan, JagadeeswaranRajendiran, RicoMagsipoc, CeciliaChoy, IvoDinov
• (SCI) DavidBrayford, dav@sci.utah.edu, Martin Cole, David Weinstein

Action Items - to establish the foundation for integration of CCB Pipeline and SCIRun tools:

1. We’ll get LONI UNIX/Grid accounts for Martin Cole and David Brayford (SCI)
2. SCI team will build 1-2 visualization apps and several image-processing filters (from the SCI suite, remember that we have ITK/VTK already installed locally) on the Grid.
3. CCB will test and report on the functionality, usability and efficiency of the SCI Viz tools (various folks here will try these viz tools & image-processing wrappers on our data.
4. CCB will build XML pipeline descriptions of few SCI processing tools and make them available as part of the Pipeline Tool Ontology/Lib.
5. We may bring several SCI people to CCB/LONI later (Nov/Dec/Jan) to help iron out remaining issues and exchange ideas in a face-to-face meeting.
6. The goal is to demonstrate by April’07 a realistic utilization of cross-center tools in our neuroimaging research efforts.

• FunctionPipeline
  • AddFunctionAssignment
  • ModifyFunctionAssignment^?

• FunctionSingleton
  • AddArgumentAssignment^?
  • ModifyArgumentAssignment^?

-- MichaelJPan - 25 Mar 2005

• ModifyExistingPipelineModule
• ModifyExistingSingletonModule

• AddNewClass^?
• ModifyExistingClass^?

• AddNewResource
• ModifyExistingResource

• AddNewFunction^?
• AddExistingFunction^?
• ModifyExistingFunction

• CreateNewModule
• AddExistingModule
• ModifyExistingSingletonModule

• ConnectingModules

• ClusteringModules

-- MichaelJPan - 25 Mar 2005

To modify a value of an existing resource (Workflow -> Data)

1. Go to "Workflow" menu
2. Click "Data"
3. Select desired variable
4. Click "Edit" Button

Activating a parameter

To activate a parameter

1. Right click on an the module
2. Select "activate"--> Select parameter to be activated

Deactivating a parameter

To deactivate a parameter

1. Right click on a parameter
2. Select "deactivate"

Binding a variable

To bind a variable to an argument

1. Click on the module in the graph
2. Click on the pane which holds the module information
3. In that pane, select the argument to bind
4. In its resource value, select the resource to bind it to

You can right click on the argument tab in the graph to make sure that the value has been bound

To add/edit additional information

1. Right click on the module
2. Click on tab titled "Edit Function"

1. Select the parameter to be modified from list

1. Enter/edit desired information

• Remember to save changes by clicking on "Workspace" and selecting "Save" Then Select Path for the file.

• Module types
  • CompositeModules
  • SingletonModuless^?- Singletons wrap around a single Function. They refer to this function by its ID.
• Module argument assignments- The values of the Function arguments are populated by variable assignments.

   <Module rdf:resource="simplealignGrid">
      <Module rdf:resource="Crop0">
         <Assign rdf:resource="CropVolume">
            <owl:allValuesFrom rdf:resource="CropVolume" />
         </Assign>
         <Assign rdf:resource="CropPadding">
            <owl:allValuesFrom rdf:resource="CropPadding" />
         </Assign>
         <Assign rdf:resource="CropCroppedVolume">
            <owl:allValuesFrom rdf:resource="CropCroppedVolume" />
         </Assign>
         <Assign rdf:resource="CropCLIOption4">
            <owl:allValuesFrom rdf:resource="CropCLIOption4" />
         </Assign>
         <Assign rdf:resource="CropTransformation">
            <owl:allValuesFrom rdf:resource="CropTransformation" />
         </Assign>
      </Module>
   </Module>

A ModuleErrorPane dispalys a table that contains errors for a given module.

-- RonNor - 01 Feb 2006

ModuleInfoPane displays information about a module when it is selected.

-- RonNor - 01 Feb 2006

A ModuleProcessPane displays a table that contains the children processes of a module.

When the user double-clicks on a row in this table, a ModuleCanvas is added as a tab to the PipeCanvas containing this pane with a call to PipeCanvas.setData(). If the data for the ModuleProcessPane is a Pipeline, a VirtualPipelineCanvas is created. If the data is a Singleton object, a ProcessCanvas is created.

-- RonNor - 01 Feb 2006

ModuleSyntaxUpdate: To Do List

GUI Changes

• Edge Drawing
  • Current Implementation
    • After modules are drawn on the graph, edges are added with PipeGraph.addEdges()
      • Iterates over PipeCells in graph and uses Module.getOutputConnections() to draw connections
    • Connections are based on arguments that are bound to a common variable in the document
  • New Implementation
    • Change PipeGraph.addEdges() so that edges are based on the graph's underlying FunctionPipeline's ParameterConnections
    • PipeGraph.addEdge(PipeCell, PipeCell, String) should be changed to PipeGraph.addEdge(ParameterConnection) instead
      • Given a ParameterConnection and a Pipeline module, can obtain an Argument bound to any Parameter that is the source or sink of a connection
• Invalidated ParameterConnections based on user action
  • Whenever a bound argument is saved in the GUI, it calls Argument.config()
  • Argument.config() should validate whether its new binding maintains the integrity of any ParameterConnections that might be associated with its Parameter

-- RonNor - 27 Jan 2006

The ModulesPanesHolder contains a tabbed pane. Tab panels include one for the "Working List", one for the "Library", and one for the "User List" if the user has provided modules in the ~/.pipeline/modules directory.

When the ModulesPanesHolder is constructed, it creates tabs for items in the map returned by PipelineManager.getModuleLists(). The keys in the map represent the name of the list, and the values are a list of modules.

The modules contained in each list are displayed in each tab as a JTree. The child nodes of the root correspond to modules. Each module is a root module within a Pipeline document. The children of these nodes are variables, classes, icons, and functions contained within the document.

A tab is also created for the "Library". This tab holds a tree that displays a categorized view of the packaged ontology functions.

At the bottom of the panel is a search bar. This bar will search the selected tab for items that match within the tree view. Matching items are highlighted.

ModulesPanesHolder supports contextual popup menus. These popup menus provide functionality for copy/pasting modules, functions, and resources.

The ModulesPanesHolder has drag-and-drop support. It utilizes the PipelineTransferHandler to create Transferable objects and to access the system clipboard.

-- RonNor - 02 Feb 2006

Many to one value binding

The Pipeline will bind multiple values to a single argument on your behalf. To specify multiple values for an argument, edit the variable bound to the argument. The simplest way to do this is to double click on the argument in the graph view of the pipeline.

Figure 1. Inserting/removing multiple value tabs for many-to-one value binding

In the "Edit variable" dialog, the "Insert value" and "Remove value" buttons near the bottom will create new tab panels for values and remove them respectively. Clicking on the insert button will insert a tab immediately after the currently selected value tab. The order of the value tabs corresponds with the order of values.

Figure 2. Resulting overall value of the argument after many-to-one value binding

List binding

To bind multiple iterations for an argument (implying multiple iterations for the argument's module and possibly other portions of the pipeline, depending on context), Pipeline provides a list binding mechanism. The "Add List Reference" button will add a combo box to the value tab and fill the combo box with all the Pipeline List or Pipeline List File variables that are available in the current module file. Selecting a list variable reference will automatically create multiple iterations for each item in the list. If there are no list variables available in the current module file, it is possible to CreatePipelineLists.

List file binding

If a Pipeline list file is available, you can specify a list URL to bind multiple iterations for an argument based on the values in the aforementioned list file. Click on "Add List URL" to specify the location of the list file. The "Select URL" button will open a "Construct URL" dialog box. Use this interface to specify the location of the list file.

Namespaces are defined in the document's root element

  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
           xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" 
           xmlns:owl="http://www.w3.org/2002/07/owl#">

Communications with a Pipeline server is accomplished with the use of TCP sockets. There are 3 parts of the total packet structure.

• The first 4 bytes of a packet specify the size of the header section.
  • [0x00, 0x00, 0x00, 0x01] -> header has 1 byte
  • [0x00, 0x00, 0x01, 0x00] -> header has 256 bytes
• The PipelinePacketHeader
• The PipelinePacketAttachment

EstablishPipelineConnection describes the process in which a connection with a Pipeline server is established.

Once the connection (and associated encryption) has been established, messages are communicated using Pipeline packets. In every packet, the following info is required:

• User login and password (every packet is authenticated)
• ... anything more?

PipelineConnectionPackets lists the packets used to communicate messages.

Note: The Pipeline currently uses its own packet structure, which is very SOAP-like. Migration of this packet structure to SOAP will be completed shortly.

To enable the Pipeline to make use of external protocols, see ImplementingAdditionalConnectionProtocols.

-- MichaelJPan - 22 Mar 2005

At this stage, the server will authenticate the user information.

If authenticated, the server will send back a SuccessfulAuthenticationPacket^?.

If not authenticated, the server will send back an UnsuccessfulAuthenticationPacket^?, and disconnect.

-- MichaelJPan - 21 May 2005

Both sides of a connection need to inform the other side once they have established the encryption ciphers, so that encrypted communications can begin. The following is an example of such a packet

<header>
   <user>${USER}</user>
   <password>${PASSWORD}</password>
   <source_host>127.0.0.1</source_host>
   <source_port>36447</source_port>
   <destination_host>127.0.0.2</destination_host>
   <destination_port>36447</destination_port>
   <type>cipherEstablished</type>
</header>

Note: This packet includes the user's credentials. That is because this packet is the first packet that is encrypted, which begins immediately once the encryption ciphers have been established.

-- MichaelJPan - 23 May 2005

This is an example of a DiffieHellman parameters packet

  <header>
  <source_port>45428</source_port>
  <destination_port>45428</destination_port>
  <encryption_algorithm>AES</encryption_algorithm>
  <encryption_strength>128</encryption_strength>
  <dh_parameters>
   161409185729581153775647009132494411332525
   879318039955946978758100904429530726656032
   187867234343861292704739684762474760744077
   916200898002702304048724147499135998295672
   865091692356970835259324103735272680100194
   577447869930237579415630488401405676518644
   792867488670859818910681080942777774432618,
   647205257076356129241205590406139634024305
   200489578884211385528076340528034701913631
   854842037175641750450734144798164286322203
   407910043637015952185386595682250146611951
   958820358558607861492737641409752570682334
   778142953440979938567634779183850321181098
   863386732757388080881773070003051466137339,
   1023
  </dh_parameters>
  <type>DHParameters</type>
  <source_host>127.0.0.1</source_host>
  <destination_host>127.0.0.2</destination_host>
</header>

DiffieHellman parameters contain 2 integers, p and g. However, one will notice that a third number is sent over. This number, used in the constructor of DHParameterSpec, specifies the size in bits of the secret key to be established.

Additionally, the packet establishes the shared secret key algorithm (in this case, AES) and the size of the key (i.e. 128).

-- MichaelJPan - 22 Mar 2005

Once the Pipeline application (client or server) has received the NetworkingAPIDiffieHellmanParametersPacket, it will use these parameters to establish its private and public key pair. The private key s is chosen from the set of integers, while the public key is generated as g^s mod p. Once the asymmetric keys have been established, the public key is then sent over to the other side in a NetworkingAPIPublicKeyPacket.

-- MichaelJPan - 22 Mar 2005

Upon connection, the server (the one who accepted the connection) will send over a NetworkingAPIDiffieHellmanParametersPacket to the client (the one who initiated the connection). These parameters will be used to NetworkingAPIEstablishAsymmetricKeys.

A client will do nothing until it receives the NetworkingAPIDiffieHellmanParametersPacket from the server.

-- MichaelJPan - 22 Mar 2005

Once the other end's public key has been received, and one's own asymmetric keys have been established, the DiffieHellman algorithm is used to generate the shared secret key k independently. The Pipeline defaults to using AES as the encryption algorithm, though it can be configured to another algorithm. Once the shared secret key is calculated, all further communications are encrypted.

-- MichaelJPan - 22 Mar 2005

This step simply waits for the other end to send over their public key, in a NetworkingAPIPublicKeyPacket.

-- MichaelJPan - 22 Mar 2005

First open a tcp socket.

-- MichaelJPan - 22 Mar 2005

This is a packet that requests a file

<header>
   <type>file</type>
   <message>request</message>
   <user>${USER}</user>
   <password>${PASSWORD}</password>
   <source_host>17.0.0.1</source_host>
   <source_port>37308</source_port>
   <destination_host>127.0.0.2</destination_host>
   <destination_port>37308</destination_port>
   <data>
    <File>
     <Values>
      <Value>
       <string><![CDATA[${FILEURL}]]></string>
      </Value>
     </Values>
    </File>
   </data>
</header>

-- MichaelJPan - 23 May 2005

This is a packet that encapsulates a part of a file

<header>
<type>file</type>
<message>send</message>
<user>${USER}</user>
<password>${PASSWORD}</password>
<destination_host>127.0.0.2</destination_host>
<destination_port>1096</destination_port>
<source_host>127.0.0.1</source_host>
<source_port>1096</source_port>
<source_filepath>${FILEURL}</source_filepath>
<attachmentPosition>0</attachmentPosition>
<attachmentLength>65536</attachmentLength>
</header>

To specify that a packet is the last of a file

<attachmentEnd>true</attachmentEnd>

is also included as part of the header

-- MichaelJPan - 23 May 2005

The server, immediately after the NetworkingAPICipherEstablishedPacket from the client, uses the credentials provided by the client in that packet to authenticate the client. If the client is authenticated, it will send over a packet specifying to the client that it has been authenticated. This header looks like this.

<header>
   <type>loginSucceeded</type>
   <user>${USER}</user>
   <password>${PASSWORD}</password>
   <source_host>127.0.0.1</source_host>
   <source_port>36532</source_port>
   <destination_host>127.0.0.2</destination_host>
   <destination_port>36532</destination_port>
</header>

-- MichaelJPan - 23 May 2005

Command packets allow a client to specify actions for the server to complete on its behalf. Some of these actions include

• load
• execute
• pause
• resume
• reset
• abort
• close

The following is an example of a

<header>
<type>processCommand</type>
<message>execute</message>
<source_host>127.0.0.1</source_host>
<source_port>1096</source_port>
<destination_host>127.0.0.2</destination_host>
<destination_port>1096</destination_port>
<user>${USER}</user>
<password>${PASSWORD}</password>
<data>${MODULE_DATA}</data>
<id>${MODULE_PATH}</id>
<digest>${MODULE_DIGEST}</digest>
</header>

-- MichaelJPan - 23 May 2005

This packet allows for the server to inform the client of the status of workflows it is running on behalf of the user.

An example of such a public key packet.

<header>
  <source_port>45428</source_port>
  <destination_port>45428</destination_port>
  <public_key>
   308201a63082011a06092a864886f70d0103013082
   010b02818100e5dab412c80776a5dd2c2d51ea6b97
   2aecc55372a0b25227bae3dd3f6f5f46ca417b1e16
   a1716f0d9fa02eba08f01312b9f017d880e08da7b4
   144202c81ea9e3b0804227d2f09ec28abe1ecacf82
   7ba72aa63f206efe7fa4c718e8ab832cf0800863cc
   aa8de1fd9313aeadbbd69ce03391c9962bb3df3572
   6653b43e99ec01b50281805c2a43338a9dadbf860b
   5b96f38d580f1ecb429912fcd11ae1f5f5a2812a56
   ece2597c002c9172dabcf866cc04bf808cd382d912
   27ce2039e8d19ee319096747bf2712403f3f3fe081
   73f5316f8de600407a906233c309d26958dc3e04ca
   f22d71e9aa0e05ec0e3d93644bf2feeec8c1ae36b6
   1004346255b733e427807e365020203ff038185000
   2818100eae1543a2d7538c1e0ee778f011cd01749c
   465a31eceede175eb507b21995a4ddf1f232911e54
   410fd9526719d59a72e43d052399f8859918dd0306
   bda15efe1ed798398a3e38e03ed371935b14f4f5d8
   9853301b82b311f273a61fcd8ebf1923e827360cc4
   15a9c3e08e166646848eadd1fdf65fb94e06b17e35
  </public_key>
  <type>publicKey</type>
  <source_host>127.0.0.1</source_host>
  <destination_host>127.0.0.2</destination_host>
</header>

-- MichaelJPan - 22 Mar 2005

Once the connection has been established, packets queued to be sent to the user while the user is logged out is sent.

Once the user has authenticated with the user, the server will send over information about the permissions the user has on the server in a NetworkingAPIUserPermissionsPacket.

This packet provides the client with information about the permissions that the user has on the server.

If the client has jobs on the server from a previous session, the server will send over a NetworkingAPIWorkflowsSynchronizationPacket

This packet allows for a client and a server to synchronize on the workflows that the server may be executing on behalf of the user from a previous session.

Nightly builds

If you are an internal user, you can grab a copy of the nightly build from

/loni/mjpan/pipeline.builds/nightly/

on an internal LONI machine which mounts the /loni file system

Class definitions in the ontology distributed with the Pipeline application

• ClassDefinitions.owl:
• FunctionDefinitionsIndex

Class: edu.ucla.loni.pipeline.gui.Options

The Options object manages user interface options. Upon construction, it loads default option values, and tries to overwrite them with any specified user options, which are stored in a file in ~/.pipeline/config/UIConfig.xml.

It provides the get/set methods for the different option settings.

• Current Options
  • OPTION_SCROLL_FACTOR
    • A multiplicative coefficient for determining the amount of pixels panned when the user performs a canvas drag and pan operation.
  • OPTION_RECENT_MODULES_COUNT
    • The number of recent modules to track
  • OPTION_HIGHLIGHT_POSSIBLE_CONNECTIONS
    • Whether or not to highlight argument cells that can be connected to a selected argument cell. This option slows down the GUI.
  • OPTION_ANIMATE_POSSIBLE_CONNECTIONS
    • Whether or not to animate argument cells that can be connected to a selected argument cell. This option slows down the GUI.
  • OPTION_ENABLE_CONNECT_TO_PARAMETER_MENU
    • Enables context popup menu for connecting arguments. This option slows down the GUI.
  • OPTION_ENABLE_VIEWABLE_ARGUMENT_FILTERING
    • Enables filtering of viewable arguments. When filtered, the menu items that are not ready to be viewed are disabled. This determination is computationally intensive.

-- RonNor - 31 Jan 2006

Bugs

• Mantis #1306
• Mantis #1311

Documentation

• FunctionFactory^?.java
• Difference between cardinality and iteration
• Client Server communication flow (setup, status updates etc) - concepts linking to the code fragments
• Need for the execution monitors hierarchy, need for sharing the hierarchy with the data model (eg: AbstractDataMonitor^?, AbstractMonitor^? are related etc)
• Explanation of Data Model, Functions, CompositeFunctions^? (concepts, code)
• All steps in validation (concepts, code)
• Steps needed to break the convoluted dependency between data validation and execution (concept, code)
• NAMIC: ** Lauren's project * Her requirements * Issues faced * Fixes made in the past two months
• Need for URLs, own locking mechanisms in URLs
• Need for locking InputArguments^?
• Need for locking OutputArguments^? when the individual processes are independent
• Steps needed to remove Encryption from the client-server communication
• Just like connections were not fixed between modules: similarly in the execution model, the flow from one module from the other does not happen - steps needed to accomplish this -

-- JonMorra - 01 Mar 2006

The way that data is passed around in all of the classes that extend AbstractGUI have now changed
Now everything that extends AbstractGUI takes in a Map with some basic parameters. Every AbstractGUI must have a Container and a GUIComponent, the same as before but now they are in a map with keys AbstractGUI.CONTAINER and AbstractGUI.COMPONENT. For example,
java.util.Map inputs = new Hashtable();
inputs.put(AbstractGUI.CONTAINER, dialog);
inputs.put(AbstractGUI.COMPONENT, getFrame());

Then, every class that extends AbstractGUI can define other things that its input map must have. For instance every EditModuleGUI has a module as inputs. So, in addition to the above code you would have to add the following in order to properly create an EditModuleGUI
inputs.put(EditModuleGUI.MODULE, module);
EditModuleGUI emg = new EditModuleGUI(inputs);

Whenever creating a new GUI class make sure to follow the convention of having all of the parameters that the GUI needs as public final static Strings as class variables. Therefore, whenever someone is trying to create this new GUI, they know what it needs (in addition to the requirements of all parent classes).

The reason behind this change was to use the createObject method in edu.ucla.loni.Pipline so that the DialogManager could handle the creation of all possible GUIs.

The are the classes in the Pipeline that have been defined for persistence so far

• UserAccount (not in Hibernate yet)
• Group
  • RootGroup
  • UserGroup
  • GenericGroup
• Predicate
  • Or
  • And
  • Not
  • ContainsFile
  • FilePermission

PipeCanvas extends a CloseableTabbedPane. A PipeCanvas is created with a root Pipeline module. There is a one-to-one match between modules in the "Working List" and existing PipeCanvases. Therefore, GUI objects and states that are specific to a loaded module instance should be kept in PipeCanvas.

The setData() method of PipeCanvas creates a ModuleCanvas or sets the selected tab to be an already existing ModuleCanvas. The type of ModuleCanvas created depends on the type of the object being set. Possible canvas types include PipelineCanvas, SingletonCanvas, ProcessCanvas, and VirtualPipelineCanvas.

For PipelineCanvas components, the PipeCanvas also keeps track of their GlobalViewGraph objects.

PipeCanvas also provides a reload() method for reloading the contained PipelineCanvas components.

PipeCanvas holds state information for the run panel buttons and menu items. There are methods to set the states and obtain the states of whether the reset, run, pause, or stop buttons should be enabled.

Note : The ModuleCanvas interface has a misleading name. Although there are a couple of implementations of the interface that exist for modules (PipelineCanvas, SingletonCanvas), other implementations do not. These include ProcessCanvas and VirtualPipelineCanvas. A ModuleCanvas was intended to represent any component that could be set as a tab in PipeCanvas, and could also have containing data refreshed using the update() method.

-- RonNor - 01 Feb 2006

A PipeCell represents a module in a PipeGraph.

PipeCells are a persistant object model. A PipeCellView represents a view of a PipeCell. The PipeCellView is more volatile. Refreshing a PipeGraph will create new PipeCellView objects for each PipeCell.

The user object associated with the PipeCell determines the color of a PipeCell. A Pipeline module will display as yellow, while a Singleton module will display as blue. The user object also determines how arguments are added to the PipeCell.

A PipeCell provides status information for it's underlying data. This is most obviously displayed as italicized text to indicate the underlying monitor status string. It also provides status information with it's status color (getStatusColor()). This is rendered as a colored outline around a PipeCell.

PipeCell holds size and location information that is used for rendering. The getPreferredSize() method calculates a preferred size for the PipeCell. This calculation takes into account the number of ArgumentCells attached to the PipeCell, whether or not the PipeCell has an icon specified, and the maximum pixel width of a status string.

For automatic cell positioning, PipeCell has a toDot() method to output the cell in the DOT language.

-- RonNor - 01 Feb 2006

PipeCellView represents a view of a PipeCell.

It has methods to determine its sizing and placement information based on information provided by its PipeCell.

A PipeCellRenderer is used to actually render the PipeCellView in the PipeGraph.

-- RonNor - 01 Feb 2006

A PipeEdge represents a connection in the PipeGraph.

Like PipeCells, PipeEdges are a persistant object model. A PipeEdgeView represents a view of a PipeEdge.

PipeEdges support highlighting with the getHighlightColor() and setHighlightColor() methods.

The graph edge routing algorithm is implemented in an inner class of PipeEdge, called DefaultRouting.

-- RonNor - 01 Feb 2006

PipeEdgeView represents a view of a PipeEdge.

A PipeEdgeRenderer is used to actually render the PipeEdgeView in the PipeGraph.

-- RonNor - 01 Feb 2006

PipeFrame is a Pipeline extension that is used for GUI management. It extends the AbstractComponent class and runs on its own thread. PipeFrame processes events that the GUI is interested in. It also provides many of the methods that are triggered by the application menus.

• getPipelineGUI() : Holds main GUI components
• getDialogManager() : Creating/grabbing dialogs

-- RonNor - 30 Jan 2006

PipeGraph is a graph representation of a Pipeline. Child modules are added as PipeCells. Module arguments are represented with ArgumentCells. Connections between modules are represented as PipeEdges.

PipeCell, ArgumentCell, and PipeEdge objects are a persistent representation of graph cells, with respect to the PipeGraph. The cells have their own respective CellViews, which are more volatile. PipeCellView, ArgumentCellView, and PipeEdgeView objects are recreated whenever the PipeGraph is reloaded.

Most user interface interaction with the graph is handled in the PipeGraphUI class.

Drag-and-drop is supported with the PipeGraphTransferHandler.

Keyboard listening is handled by the PipeGraphKeyListener.

A PipeGraph supports zooming with the setScale() and getScale() methods.

Exporting the graph as an image (a screenshot) is supported with the getImage() method.

Automatic graph layout is supported in two ways. If modules do not contain any positioning information, they will be laid out using an algorithm implemented in the calculateAutoPosition() method. It is also possible to calculate module positioning based on an external graph layout engine. This is done by obtaining a DOT representation of the graph and submitting it to the graph server defined by PipeGraph.GRAPH_LAYOUT_SERVER. Use the formatFromDot() method to automatically reposition the modules using DOT.

-- RonNor - 01 Feb 2006

The PipeGraphKeyListener listens for key events in a PipeGraph.

• Key events handled
  • Up/Down/Left/Right
    • "budges" a PipeCell in the corresponding direction
  • Delete
    • Removes the selected PipeCell and module from the Pipeline

-- RonNor - 01 Feb 2006

The PipeGraphTransferHandler is used to accept a drop in a drag-and-drop operation. The types of objects that can be dropped into a PipeGraph are Module, Function, and Resource.

-- RonNor - 01 Feb 2006

PipeGraphUI handles most user interaction with a PipeGraph.

It contains methods that paint the background grid and LONI logo to the PipeGraph.

PipeGraphUI is mainly used to handle mouse events, either through the RootHandle or MouseHandler.

Mouse actions that occur on a CellView will occur on the RootHandle. Currently, RootHandle is used to handle creating connections via a drag between ArgumentCells, ctrl+drag PipeCell cloning, and updating the underlying data of the new coordinate positions of PipeCells.

Other mouse events are handled by the MouseHandler. The bulk of this inner class is devoted to creating contextual popup menus. It is also used to highlight edges, and connectable cells when an ArgumentCell is clicked on.

-- RonNor - 01 Feb 2006

The LONI Pipeline Processing Environment is hosted at UCLA's Laboratory of Neuro Imaging. It has been used quite successfully since its release in the neuroimaging field. It has been downloaded over 700 times and there are close to 100 registered users within LONI and its collaborators. The Pipeline has sped up research in this area, allowing researchers to focus upon the research and not the mundane and repetitive administrative tasks of data management and processing. It also has provided an easy mechanism for researchers to disseminate their research processes.

All software grow and evolve along with user feedback, and the Pipeline is no different. Since its initial release, much of the feedback has been constructive, suggesting useful functionalities that could be integrated with the application. We have added features and removed bugs accordingly. The architecture of the environment has been reorganized to aid in accomplishing these, and future, tasks.

A modular architecture, with task specific components that provide quick and easy customization, is currently under development. By cleanly separating these components, the development process has reduced in complexity and speeded up dramatically. Taking advantage of this has allowed us to make other dramatic improvements (which are included in the upcoming v3 release). These new improvements include improved security, grid computing integration, automated process management, automated resource management, advanced debugging, and fault tolerance.

Integration of additional functionalities, which include vizualization, validation, and provenance, are planned. For developers who wish to make their own contributions to the development of the Pipeline, an application programming interface (API) will be made available.

PipelineCanvas is a ModuleCanvas that contains a ModuleProcessPane, ModuleErrorPane, and a scrollpane that holds a PipeGraph.

The ModuleProcessPane and ModuleErrorPane are contained within a modulePane, which is hidden in a non-expanded state initially. When the PipelineCanvas is updated, it updates its ModuleProcessPane and ModuleErrorPane also. If any of the tables contained within those panes gets populated with data, the modulePane is expanded with a call to showModulePane().

PipelineCanvas also contains a graph view of its root Pipeline. The constructGraph() method builds a graph from a Pipeline and adds modules to it.

-- RonNor - 01 Feb 2006

The following people and instututions have collaborations with the Pipeline

• Center for Computational Biology (CCB)
• Biomedical Informatics Research Network (BIRN)
• National Alliance for Medical Image Computing (NAMIC)
• Scientific Computing and Imaging Institute (SCI Institute)

This is a list of the packets that are used to communicate messages

• NetworkingAPIDiffieHellmanParametersPacket
• NetworkingAPIPublicKeyPacket
• NetworkingAPICipherEstablishedPacket
• NetworkingAPILoginSucceededPacket

• NetworkingAPIWorkflowsSynchronizationPacket
• NetworkingAPIUserPermissionsPacket

• NetworkingAPIProcessCommandPacket
• NetworkingAPIProcessEventPacket

• NetworkingAPIFileRequestPacket
• NetworkingAPIFileSendPacket

-- MichaelJPan - 23 May 2005

Current contributors

• Principal investigator
  • Arthur W Toga
• Technical lead
  • MichaelJPan
• Programmer
  • JagadeeswaranRajendiran
    • Liaison to NA-MIC
    • Main responsibilities: configuration, execution, scheduling
  • Jia Wei Tam
    • Main responsibilities: stability and scalability issues
  • Arash Payan
    • Main responsibilities: overall usability
  • Jon Morra
    • Main responsibilities: Matlab & R integration, GUI framework
• Undergraduate student
  • StephenTabaquin

Past contributors

• DavidRex (Technical lead)
• Jeffrey Q Ma (Programmer)
• RonNor (Programmer)
• Heather Malinick (undergraduate student)
• LaurenOliver (undergraduate student)
• MiraeYang (undergraduate student)
• SheenaKamra (undergraduate student)

Pipeline

Download the latest release of the Pipeline, and view the release notes for more information.

How can I help to improve Pipeline?
The best way to help out is by using Pipeline and giving us feedback. You can sends comments/suggestions to us by email at pipeline{at}loni.ucla.edu and submit bug reports to us through our bug tracker.

What license is Pipeline release under?
Pipeline is release under the LONI license. You can view a comparison of the LONI License with other licenses as well.

Who funded the development of Pipeline?
A list of funding agencies can be found here.

Who is currently using Pipeline?
Many people in our lab and in other labs are currently using Pipeline.

The LONI Pipeline v3 has the following features

• Modular architecture
• Server/client communications via TCP
• AES encrypted communications
• Ontology enabled data processing using OWL
• Runtime processing of filenames
• Automatic cache management
• Grid engine integration
• File streaming
• URL wrapper and data resource abstraction
• Comprehensive logging
• Process stream capturing
• Integrated viewer
• User friendly GUI
  • Tooltips
  • Consolidated messaging
  • Node connection highlighting
  • Multiple node operators
• Runtime list processing
• Bash shell script output
• Transparent module syntax updater framework
• Documentation via Wiki, Doxygen, and Javadoc

The PipelineGUI holds references to the main menu, main panel, root JFrame component, and PipelineListener.

• Main menu
• Main panel
• PipelineListener

-- RonNor - 31 Jan 2006

• Function - something that takes a set of inputs and returns a set of outputs. (technically, programs dont return anything... all the outputs are side effects, but thats a small detail for now)
• Parameter - an input to, or output of, a function
• Module - an instantiation of a function
• Resource - an instance of any class
• Variable - resources that can be arguments to a module.
• Argument - a variable that is bound to a parameter in a module

Pipeline Development Group Action Items

2008

December

December, 2008, Week 1

December, 2008, Week 2

---+++ Back to main Pipeline Wiki

PipelineListener listens for ActionCommands and calls the appropriate action methods for a command.

It is also registered as a WindowListener for the root frame. Currently, it listens for a window closing events.

-- RonNor - 31 Jan 2006

PipelineModules are XML descriptors of program usage. These descriptors provided information about how a program is run, so that the Pipeline can correctly execute them on your behalf.

The Pipeline client provides a set of LibraryModules. If you are using programs not located on the LONI hosted server, you can create your own module.

For those who are interested, PipelineSyntax documents the syntax of PipelineModules

The following are some examples of PipelineModules. Click on any one for more information

• NAMIC
  • InstallITKModule
  • InstallVTKModule
  • ITKExercisesModule
  • ITKMultiModalityRigidRegistrationModule
  • UNCShapeModule

• LONI
  • FirefoxModule
  • SimpleAlignModule
  • FreesurferTutorialModule

The packet attachment is a binary data, of a length specified in the header. In order to provide for file streaming, the length of this attachment is limited to 65536 bytes (64kb). That means, a file larger than this size will need to be split up among many packets.

The PipelinePacketHeader specifies meta-data about the particular packet. Some of this info includes

• the size of the attachment
• the encryption of the attachment
• source ip information
• destination ip information
• user login and password (if encryption has been established)

The packet header for each packet type can be found in the description for each packet type, listed in PipelineConnectionPackets

-- MichaelJPan - 01 Jun 2005

User Links & Demos

• Quick Guide (.pdf)
• Creating a Basic Pipeline (.mov)
• Creating a list for Multiple Inputs (.mov)
• Steps Required for Running a Pipeline on the LONI GRID (.mov)
• Grid Execution (.mov)
• Simple Copy Definition Creation (.mov)
• More Complex Definition Creation (.mov)
• Advanced Parameter Options (.mov)

Developer Links

• Developer Documentation
• Meeting Minutes
• Human Interface Guidelines
  • Apple OS X
    • Apple OS X Java Reference Library
  • Gnome
  • Java
  • KDE
  • Windows
• Taxonomy of Graphical Workflow Environments

Think you've found a bug in Pipeline? We'd like to hear about it. Head on over to our bug tracking page and let us know, so we can work to improve the Pipeline for everyone.

PipelineModules are stored in XML.

There are currently 2 iterations of the XML DTD for PipelineModules.

• PipelineV1Syntax^?
• PipelineV3Syntax

-- MichaelJPan - 02 Feb 2005

A PipelineTransferHandler is used by the Pipeline application to create Module, Function, or Resource Transferable objects for drag-and-drop operations. It also provides support for accessing the system clipboard.

This is used in the ModulesPanesHolder to drag/drop objects to a PipeGraph.

-- RonNor - 01 Feb 2006

PipelineModules in Pipeline v3 are stored in OWL, which is a recommended specification of the World Wide Web Consortium (W3C). The Web Ontology Language for Services (OWL-S) is a good starting point for the description of the necessary knowledge constructs for describing workflows. The OWL-S submission to the W3C for becoming a W3C recommendation can be found here.

The following explains some of the technical terms necessary for discussing pipeline module descriptions.

• Module metadata
  • NamespaceDeclaration
  • VersionInformation
• Namespaces
• ClassDefinitions
• FileDependency

• ResourceDefinitions
• ClassInstances
• FunctionDefinitions
• ModuleDefinitions

The DTD of the documentions should go ->here<-

Past presentations

The following is a list of presentations of the Pipeline

• pipeline.20041210.ppt: This was presented to NAMIC collaborators at the NAMIC Toolkit dissemination meeting in Boston, held at the Surgical Planning Lab (SPL) at Harvard's Brigham and Women's Hospital (BWH), 10 December 2004.
• pipeline.20050217.ppt: This was presented to UCLA Mathematics collaborators as part of a Neuroscience training course, held at UCLA's Institute for Pure and Applied Mathematics (IPAM), 17 February 2005.
• pipeline.20050218.ppt: This was presented to NIH Science Officer John Haller for the Center for Computational Biology (CCB) Science Advisor meeting, held at UCLA's Brain Mapping Center, 18 February, 2005.
• pipeline.20050221.ppt: This was presented to NAMIC collaborators at the NAMIC All Hands Meeting, held at the Hotel Monaco in Salt Lake City, 21 February 2005.
• pipeline.20050428.ppt: This was presented to the Scientific Review Board at the P41 (LONI Resource) Site Visit Progress Review, held at UCLA's Brain Mapping Center, 28 April, 2005.
• pipeline.20050526.ppt: This was to be presented to the ScientificAdvisoryBoard as part of the progress report for year 1 of the U54 grant establishing CCB as one of the 4 national NCBC Centers, held at UCLA LONI, 26 May 2005.
• pipeline.050601.ppt: This was presented to MouseBIRN AHM, held at UCLA LONI, 1 June 2005.
• pipeline.20050629.ppt: This was presented at the NAMIC Programmers' Retreat at MIT, 29 June 2005.
• pipeline.20051103.ppt: This was presented at the CCB Principals Software Development Meeting at LONI, 3 November 2005.
• pipeline.20060109.ppt: This was presented at the NAMIC All hands Meeting, held at the Hotel Monaco in Salt Lake City, 9 January 2006.
• namicEAB.20060110.ppt: This was presented at the NAMIC EAB meeting, held at the Hotel Monaco in Salt Lake City, 10 January 2006.
• pipeline.20060113.ppt: This was presented at the end of the NAMIC AHM Programmers Session, head at the Hotel Monaco in Salt Lake City, 13 January 2006.

Tutorial syllabus

• Tutorial syllabus: This is the syllabus of the tutorial

Templates

• CCB Powerpoint template
• LONIR Powerpoint template

A ProcessCanvas is a ModuleCanvas that holds a table of error values for a ProcessMonitor.

-- RonNor - 01 Feb 2006

Overview

ProcessScheduler schedules all submitted processes based on its 'acquired' priority. It is responsible for enqueuing processes, dequeuing processes and scheduling them by maintaining priority based queues.

Friend Classes

UserGroupManager^? and UserAccountManager^? are friend classes that help ProcessScheduler in deciding priorities for processes.

-- JagadeeswaranRajendiran - 22 Mar 2005

• ModuleSyntaxUpdateToDo
• RemoteServerAdministrationDevDoc
  • Tutorials
    • AddingNewServerCommandTutorial
    • AddingNewRemoteServerOptionsPanelTutorial
• GUI Documentation
  • PipeFrame
    • PipelineGUI
    • PipelineListener
    • PipelineTransferHandler
    • DialogManagerDevDoc
    • Options
    • MainMenuGUI
      • RecentModuleManager
    • MainPanelGUI
      • Workspace
        • PipeCanvas
          • PipelineCanvas
            • PipeGraph
              • PipeCell
                • PipeCellView
              • PipeEdge
                • PipeEdgeView
              • ArgumentCell
                • ArgumentCellView
              • PipeGraphUI
              • PipeGraphTransferHandler
              • PipeGraphKeyListener
          • SingletonCanvas
            • ModuleProcessPane
            • ModuleErrorPane
          • ProcessCanvas
          • VirtualPipelineCanvas
      • GlobalViewGraph
      • ZoomPanel
      • LogMessagesPane
    • ModuleInfoPane
    • ModulesPanesHolder
• Suggestions for improving the GUI

-- RonNor - 27 Jan 2006

The RecentModuleManager keeps track of the recent module documents that have been opened or saved recently by the user. By default, a file is saved to ~/.pipeline/recentModules maintaining a record of the recent modules. The manager is used to load from and write to this file.

getCapacity() returns the number of recent modules to keep track of. This is a user defined option that is configurable in the GUI options.

-- RonNor - 31 Jan 2006

The following is a humorous comparison of the release numbering of the LONI Pipeline v3 against various organizations

LONI Pipeline	Open source	Microsoft	Google	Apple
internal testing	alpha	1.0	shh... top secret	unsubstantiated rumors
alpha	beta	2.0	beta	rumors with possibly some substance to them
beta	release candidate	3.0	beta	developer seeds
beta	1.0	various: 3.1, 95, 98, 2000, XP	beta	release
release	2.0	SP1,2,3,...	beta	recall

Starting with version 3, the Pipeline release numbers will be assigned by the following conventions: v${version.major}.${version.minor}${release.branch}${branch.major}${branch.minor}

For example, v3.0beta2.1 specifies that this is major version 3, minor version 0, beta release, branch major 2, branch minor 1

Current development schedule:

• Client/standalone pre-alpha release (Released 30 March 2005)
• Client/standalone alpha release (Released 18 April 2005)
• Client/standalone beta release (Released 11 June 2005)
• Server beta release (Released 31 August 2005)

A note on our release number convention can be found at ReleaseNumberConvention

Upcoming features

Beyond the releases listed above, we have identified features which we would like to implement for future versions of the software.

• processing statistics gathering
• execution estimation
• recursion module
• multiplexor module
• functionmap module (this applies many functions to the same dataset, as opposed to applying the same function to multiple datasets)
• user friendly editing
  • drag&drop of data files
• self updating capabilities
• nested list processing
• analysis validation
• provenance
• automated file translations
• ensure disk space availability
• write ahead logging for crash recovery
• search capabilities
• message of the day
• advanced data viewer interface
• graduated difficulty tutorial
• explore computation as utility to be rented
• persistence for data/ProvenanceEntry
• data-centric graph

Got more items on your wish list? Add them here!

DownloadSoftware

Remote Server Administration: Developer Documentation

BetaRelease_20060120 contains a framework for remote server administration. This document outlines the design of this new component.

Design

ServerCommand and ServerResponse Framework

Remote server administration relies mostly on two extensions, a RemoteServerManager on the client and a ServerManager on the server. The RemoteServerManager issues ServerCommands and listens for ServerResponses. The ServerManager, in turn, listens for ServerCommands and issues ServerResponses.

The ServerResponse interface extends ServerCommand and they share a base implementation, AbstractServerCommand. Given this heirarchy, the ServerResponse framework is almost completely analogous to the ServerCommand framework. For brevity, this document will only outline the ServerCommand framework.

ServerCommands are generated on the client and may or may not solicit a ServerResponse in return. The ServerCommand, itself, is not submitted as an event. The ServerCommand is wrapped in a ServerCommandWrapper. This wrapper allows for multiple ServerCommands to be issued at a time. The commands will be processed in the order that they were originally added to the wrapper. The wrapper also holds information about the user issuing the command, the client host, and the server host.

Once the commands are wrapped, a ServerCommandEvent needs to be created with the wrapper. A ServerCommandEvent extends a NetworkEvent, so once the event is submitted, it is handled appropriately by the NetworkManager.

The NetworkManager will take the ServerCommandEvent and hand it off to the ServerPipelineConnection for processing. The connection extracts the wrapper from the packet. It then iterates over the commands and submits them as events, to be handled by the ServerManager. It should be noted here that ServerCommandWrapper uses ServerCommandFactory to create the proper instantiation of a ServerCommand based on the JDOM Element representations of each ServerCommand.

Remote Server Administration and the Server Administration GUI

The RemoteServerManager listens for ConnectionEvents and determines which connections qualify for remote administration. Once a session has been established, if the connection allows for administrator level access, the connection is registered by the RemoteServerManager. The RemoteServerManager will also unregister a connection if the connection has been lost.

Any initial synchronization of information that the client needs to make with the server should be handled in the RemoteServerManager.registerRemoteServerConnection() method. For example, refreshUserGroups() is called to synchronize the servers accounts with client in order to remotely manage users and groups. At some point in the future, synchronization should also occur for remotely configurable server options.

The client uses the ServerAdministrationGUI to administer remote server connections. The ServerAdministrationGUI uses a reference to the RemoteServerManager to obtain the information necessary to administer remote servers. The ServerAdministrationGUI displays a combobox that holds the registered connections that allow remote administration. When a connection is selected, the bottom portion of the GUI displays a ServerAdministrationPanelGUI. The ServerAdministrationPanelGUI contains a ButtonBar on the lefthand portion in toggle mode. Each button corresponds to a RemoteServerOptionsPanel that is displayed in the center portion of the ServerAdministrationPanelGUI. There are currently RemoteServerOptionsPanels for server ontology and user accounts. These components are used to generate ServerCommands.

Relevant Classes

• core package
  • server commands
    • edu.ucla.loni.pipeline.server.command.ServerCommand (interface)
    • edu.ucla.loni.pipeline.server.command.ServerCommandFactory
    • edu.ucla.loni.pipeline.server.command.ServerCommandWrapper
    • edu.ucla.loni.pipeline.server.command.*ServerCommand (actual server commands)
    • edu.ucla.loni.pipeline.server.command.AbstractServerCommand
    • edu.ucla.loni.pipeline.server.command.ServerCommandEvent
  • server responses
    • edu.ucla.loni.pipeline.server.response.ServerResponse (interface)
    • edu.ucla.loni.pipeline.server.response.ServerResponseFactory
    • edu.ucla.loni.pipeline.server.response.ServerResponseWrapper
    • edu.ucla.loni.pipeline.server.command.*ServerResponse (actual server responses)
    • edu.ucla.loni.pipeline.server.command.AbstractServerResponse
    • edu.ucla.loni.pipeline.server.command.ServerResponseEvent
• client package
  • edu.ucla.loni.pipeline.extensions.RemoteServerManager
  • edu.ucla.loni.pipeline.events.UserGroupRefreshListener
• server package
  • edu.ucla.loni.pipeline.server.ServerManager
• gui package
  • edu.ucla.loni.pipeline.gui.ServerAdministrationGUI
  • edu.ucla.loni.pipeline.gui.ServerAdministrationPanelGUI
  • edu.ucla.loni.pipeline.gui.RemoteServerOptionsPanel (interface)
    • RemoteServerUserGroupOptionsPanel
      • UserGroupManagementGUI/Listener
        • UserGroupGraph
        • UserGroupGraphModel
        • UserGroupGraphUI
    • RemoteServerOntologyOptionsPanel

Tutorials

• AddingNewServerCommandTutorial
• AddingNewRemoteServerOptionsPanelTutorial

-- RonNor - 27 Jan 2006

The following steps should be taken to remove a group or user in the UserAndGroupManagement dialog:

1. Right-click on the user or group node you would like to remove
2. Select the "Remove" option from the pop-up menu
3. The user or group should no longer appear.
4. If you removed a group that was the only parent group of any user or group, those child groups will now be children of the root group until you explicitly SetParentForUserOrGroup

-- MichaelJPan - 12 Sep 2005

Bugs (as well as feature requests) in the Pipeline software can be reported to our bug-tracking software Mantis, which is located here

Please feel free to add comments

-- MichaelJPan - 13 Feb 2004

The following are publications related to the Pipeline.

If you use the Pipeline in your research, please cite one of the following publications. Also available is the bibtex file.

Journal articles

• David E Rex, Jeffrey Q Ma, and Arthur W Toga. The LONI Pipeline Processing Environment. NeuroImage, 19:1033–1048, 2003. pdf

Conference papers

• Conference papers here...

Workshop papers

• Michael J Pan and Arthur W Toga. A grid enabled workflow management system for managing parameter sweep applications in neuroimaging research. Proceedings of the Fourth International Workshop on Biomedical Computations on the Grid (BioGrid06^?), Singapore. pdf

Conference abstracts

• Michael J Pan, David E Rex, Arthur W Toga. Managing parameter sweep application workflows in neuroimaging research. Proceedings of Human Brain Mapping 2006, Florence, Italy.
• Michael J Pan, David E Rex, Arthur W Toga. The LONI Pipeline Processing Environment: Improvements for Neuroimaging Analysis Research. Proceedings of Human Brain Mapping 2005, Toronto, Canada. pdf
• Arthur W Toga, David E Rex, Jeff Ma. A graphical interoperable processing pipeline. Proceedings of Human Brain Mapping 2001, Brighton, England.

Technical reports

• David E Rex. LONI Pipeline Helps Process Volumes of Data. BIRNing Issues Newsletter, Volume 1, Number 3, page 5. pdf

Working Drafts

• Michael J Pan, David E Rex, Arthur W Toga. Coordination of scientific workflows for collaborative research. pdf
• Michael J Pan, Arthur W Toga. Multi-granularity parallelization for scientific workflow management. pdf
• Michael J Pan. Disconnected operation in scientfic workflow management systems. pdf

@inproceedings{toga2001pipeline,
  author    = {Arthur W Toga and
               David E Rex and
               Jeff Ma},
  title     = {A graphical interoperable processing pipeline},
  booktitle = {7th Annual Meeting of the Organization for Human Brain Mapping},
  year      = {2001},
  pages     = {S266}
}

@inproceedings{pan2005pipeline,
  author    = {Michael J Pan and David E Rex and Arthur W Toga},
  title     = {The LONI Pipeline Processing Environment: Improvements for Neuroimaging Analysis Research},
  booktitle = {11th Annual Meeting of the Organization for Human Brain Mapping},
  year      = {2005}
}

@article{rex2003pipeline,
   author={David E Rex and Jeffrey Q Ma and and Arthur W Toga},
   title={The LONI Pipeline Processing Environment},
   journal={NeuroImage},
   volume={19},
   pages={1033–1048},
   year={2003}
}

@inproceedings{pan2006psaWorkflowGrid,
  author    = {Michael J Pan and Arthur W Toga},
  title     = {A grid enabled workflow management system for managing parameter sweep applications in neuroimaging research},
  booktitle = {4th International Workshop on Biomedical Computations on the Grid (BioGrid2006)},
  year      = {2006}
}

• IDs- Following RDF specifications, everything in a pipeline is a resource. Every resource can be uniquely identified by its ID, specified by its "rdf:ID" attribute. IDs are not meant to be human readable.
• Resource references- In order to define relationships between resources, we need a manner of referring to any resource from any other resource. This is done by using the attribute "rdf:resource".
• ResourceValue
• Labels- Every resource can have a set of labels. These labels are intended to be short, human readable names that identify the resource to a human. Labels can be localized.
• Comments- Every resource can have a set of comments. These comments are intended to provide (possibly verbose) explanations of the resources themselves. Comments can be localized.

Resource values are constructed in a complicated manner, taking into account several factors:

• host flipping - because communications between client and server is facilitated by maintaining a reference to the same Pipeline module, we co not change the text of the document. instead, the server, in memory, flips the references to the hosts, so that a reference to localhost actually points to the client, and a reference to the server's host points to localhost
• value padding - sometimes values for a resource need to be padded to its correct cardinality. This can happen when a user has not specified the value, but where one is needed. It can also happen when we need to iterate through the same data many times to accomodate for multiple parameters.
• input, output, and process cardinalities
  • a resource connected to more than one module may need to have multiple cardinality values.
• cardinality virtuality
  • sometimes the cardinality of the values of a resource need to be "virtualized", as it is possible that multiple iterations of a module will use the same values
• string composition
  • string values are literals
  • string references are references to the string values of other resources. this allows users to not have to repeat the same value if not necessary
• value filtering
• value grouping
• list binding - values for a resource are bound to a list if the attribute bindValueFromURL exists and its value is set to true.

  <CommandLineOption rdf:ID="GlobalVariableGWC_SVT_stats">
    <rdfs:label><![CDATA[GlobalVariableGWC_SVT_stats]]></rdfs:label>
    <rdfs:comment><![CDATA[global variable GWC_SVT_stats]]></rdfs:comment>
    <Values>
      <Value>
        <string rdf:resource="GlobalVariablestats_dir" />
        <string rdf:resource="GlobalVariablesubject_MRI" filter="base" />
        <string><![CDATA[_GWC_SVTstats.txt]]></string>
      </Value>
    </Values>
  </CommandLineOption>

-- MichaelJPan - 31 Mar 2005

This script will run the Pipeline using Ant, by including all the external libraries for you in the classpath.

You will have to modify some properties

• ${lib.dir}
• all the external libraries to be included

Once that is done, to run, just issue the command "ant run"

• build.xml: ant script for pipeline

This script will run the Pipeline under a bash shell, by including all the external libraries for you in the classpath.

Modify $LIB_DIR to match the directory where you have saved all the external libraries.

• run.sh: bash script for starting the pipeline application

-- JonMorra - 09 Jan 2006 Before running any Matlab commands, Matlab must be started contact your administrator for help

Running Matlab Commands

Create a new Matlab Command module and then drag it to the canvas. On the argument section type in the Matlab command that you wish to run and run the pipeline. The commands will be sent to Matlab.

Running Matlab M Files

Create a new Matlab M File module and then drag it to the canvas. Make the variable be the location of the M File. The commands will be sent to Matlab when the pipeline is run.

Use the phantom outputs to connect these modules to the rest of the pipeline. Note that Matlab must write to a file because there is no way to get the output.

• Windows - double click on the pipeline.jar file
• MacOSX^? - double click on the pipeline.jar file

• Command line (shell) - type "java -jar pipeline.jar" from the directory where the jar file is located

-- JonMorra - 09 Jan 2006

Before running any R commands, R must be started contact your administrator for help

Running R Commands

Create a new R Command module and then drag it to the canvas. On the argument section type in the R command that you wish to run and run the pipeline. The commands will be sent to R.

Running R Files

Create a new R File module and then drag it to the canvas. Make the variable be the location of the R File. The commands will be sent to R when the pipeline is run.

Use the phantom outputs to connect these modules to the rest of the pipeline. Note that R must write to a file because there is no way to get the output.

To run your own server on cerebro-sn1, using XML-based Authentication, please follow these steps:

1. Make sure your {user.home}/.pipeline/config/ exists on cerebro-sn1.loni.ucla.edu

2. Make sure the following options are set:

  
   -gridengine.enabled = true {Default)
   -value associated with gridengine.class is DRMAA {Default}
   -value associated with cache.directory is Unix_Cache {Default}
   -value associated with Unix_Cache

   - Eg: if your config option looks like this:

     <option name="cache.directory" value="Unix_Cache">
       <choices>
         <choice>
           <name>
             Windows_Cache
           </name>
           <value>
             C:\Documents And Settings           </value>  
         </choice>
         <choice>
           <name>
             Unix_Cache
           </name>
           <value>/tmp/loniuser</value>
         </choice>
       </choices>
     </option>

then, you would change value associated with choice Unix_Cache to a directory on /cxfs/ where you as a user have permissions to write into -

     <option name="cache.directory" value="Unix_Cache">
       <choices>
         <choice>
          <name>
            Windows_Cache
          </name>
          <value>
            C:\Documents And Settings          </value>  
         </choice>
         <choice>
           <name>
             Unix_Cache
           </name>
           <value>/cxfs/tmp/loniuser</value>
         </choice>
        </choices>
      </option>

3. Run on your pipeline client (pipeline.client.jar) on your local machine (eg: turrell.loni.ucla.edu)

  
       Click 'Server' -> 'Manage Users/Groups' 
       Click 'New User'
       Enter your login name: e.g. 'loniuser'
       Enter your password
       Click OK

       In the GUI, drag a line from 'admin' to your_login_name (e.g. 'loniuser' )

4. Copy the following files:    
  
   groups.xml and users.xml

   from config directory (${user.home}/.pipeline/config) on client machine to cerebro-sn1   

5. On cerebro-sn1, run the following command:

   java -d64 -jar pipeline.server.jar -port=port_number

   (e.g. java  -d64 -jar pipeline.server.jar -port=8006)

   Note:    If the server does not bind to appropriate port number, one has to build
            a pipeline.server.jar (from the sources) with the following change:

            Set network.port = port_number (in server/src/resources/config/server.xml)

-- JagadeeswaranRajendiran - 15 Mar 2006

These are the client features released along with the beta server

• consolidate ontology functions into one pane
• ask user for permission to overwrite existing files
• password encryption gui
• save before exit option

These are the server features whose testing is still be completed

• runtime information persistence
• extract network library
• job completion notification (via events within application, system to external applications, and email to users)
• standardize modules
• convert server pipelines
• document pipelines
• user and group account configuration using database
• logging to database
• Application.getApplicationResource should load the default config file, and then overwite any property specified in the user's directory. currently is exclusive.
• srb integration
• quality assurance test suite
• Pipeline-aware load balancer

These are the steps for setting up a server to run.

• Ensure SoftwareRequirements
• Define users, groups, and permissions in UserAndGroupManagement
• ConfigureServerSettings
• StartServer
• Publish access information for your users
• RunningYourOwnServer

Having problems?

• TroubleShooting
• ReportBugs

The following rules apply when setting a user or group's parent:

• The user or group must not be descended from the new parent
• The parent must be a group (green or gray node, NOT blue)
• Setting a parent for the root group is not permitted
• A user or group may have multiple parents
• A user or group must have at least one parent (with the exception of the root group, which has none)

The following steps should be taken to set the parent group for a user or group in the UserAndGroupManagement dialog:

1. Click the new parent group and drag the mouse to create a parent connector
2. Drag this parent connector over the group you would like to set
3. If certain conditions (see above) are met, the parent connector will become highlighted in red
4. Release the highlighted parent connector over the group to complete the process

To remove a parent group relationship:

1. Right-click on the parent connector you would like to remove
2. Select the "Remove" option from the pop-up menu
3. If you removed the only parent connector of any user or group, that user or group will now be a child of the root group until a parent is explicitly set again

-- MichaelJPan - 12 Sep 2005

Follow these steps

1. CreateNewModule (or CreateNewPipelineModule)
2. AddNewParameter
3. AddNewResource
4. ModifyExistingModule

or, try the ExecutionWizard.

SIGFlow meetings

• MeetingMinutes200510271300
• MeetingMinutes200511221300
• MeetingMinutes200512201300

This document provides the instructions for enabling Java WebStart^? delivery of the Pipeline processing environment.

Creating the key

to create the key to sign the jar

prompt$ keytool -genkey -alias loni -keypass pipeline
Enter keystore password:  ********
What is your first and last name?
  [Unknown]:  
What is the name of your organizational unit?
  [Unknown]:  LONI
What is the name of your organization?
  [Unknown]:  UCLA
What is the name of your City or Locality?
  [Unknown]:  Los Angeles
What is the name of your State or Province?
  [Unknown]:  CA
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=Unknown, OU=LONI, O=UCLA, L=Los Angeles, ST=CA, C=US correct?
  [no]:  y

Signing the Jar

to sign the jar, the command is

jarsigner <jarfile> <alias>

so to sign the pipeline.jar file given the above key,

prompt$ jarsigner ./build/pipeline.jar loni
Enter Passphrase for keystore: *******

This is for informational purposes only. This will be automated by the Ant build script prior to release.

-- MichaelJPan - 18 Apr 2005

Simple align module

This pipeline module aligns images to a standard.

Screenshot

Files

• simpleAlign.pm: Pipeline v1 module
• simpleAlign.module: Pipeline v3 module

A SingletonCanvas is a ModuleCanvas that displays information on child processes, module errors, and the execution status of the Singleton module associated with it.

It contains a ModuleProcessPane and a ModuleErrorPane.

-- RonNor - 01 Feb 2006

The LONI Pipeline is available under the LONI software license. This license is an open source license, similar to the Apache software license, with an additional restriction of non-redistribution. This restriction is a result from stipulations of our funding resources to track downloads and usage. The following is a comparison of commonly used open source licenses + the LONI license

Institution	License	Version	Date	Restriction(s)
MIT	MIT			License inclusion & liability disclaimer
University of California	BSD	New	22 July, 1999	Above + No endorsement of any distribution
Apache Foundation	Apache	1.1	2000	Above + Acknowledgement of creator
GNU	Lessor General Public License	2.1	February 1999	Above + inclusion of original source code
GNU	General Public License	2	June 1991	Above + inclusion of all source code
LONI	LONI	1.0	2005	Apache + No redistribution to outside organizations + no implied commercial licensing

The following are the requirements for running the Pipeline.

Pipeline application

• Java 1.4.2+

Binary executables

Please note that the LONI Pipeline is an environment for running Pipeline modules, and do not come with any external programs. Unlike other execution environments, we are capable of executing any program; it would thus be hard for us to package all programs that the Pipeline can run.

LONI hosts a Pipeline server for the LibraryModules included with the Pipeline client. If you have executing modules to be run outside the LONI hosted server, please make sure that the executables you are trying to run are available. Some common neuroimaging packages used at LONI are

• AFNI
• AIR
• MNI
• Freesurfer

Source code repository

Internal access

• Make sure you are in the following groups: hackers, piplncmt, piplndev
• server(s): any system that mounts the /loni file system
• cvsroot: /loni/src/cvsroot
• use ssh/ssh2 mode
• module: pipeline/pipeline.3.0.internal

External access (for internal users)

• server(s): autarch, inire
• all other info same as above

External access (for external users)

• coming soon

The following is a list of requirements for each class to be persisted by Hibernate

• Default constructor (i.e. no parameters)
• get/set methods for every field to be persisted
  • need to be get${Field}() and set${Field}(value)

-- MichaelJPan - 21 Jun 2005

The following are statistics about the source code of the various versions of the Pipeline, calculated by using sclc, on the latest version of the code checked out from our CVS Repository.

Legend

• Lines- total number of lines in the files
• Blank- number of blank lines
• Cmnts- number of comment lines
• NCSL - number of non-comment source lines
• AESL - ``assembly equivalent source lines'' (it calculates this using assembly-equivalence factors published by Software Productivity Research)

Pipeline v1.0 (this is not a particularly accurate detailing of v1 code, as there are duplicate directories in there)

Lines  Blank  Cmnts   NCSL     AESL     
=====  =====  =====  =====  ==========  =======================================
 2888    455    622   1838      9190.0  ----- Tcl -----  (13 files)
 1411    386    473    552      8280.0  ----- make -----  (18 files)
131088  33091  33492  70143    420858.0  ----- C++ -----  (240 files)
 2136    222   1255    872      2180.0  ----- C -----  (34 files)
  948     94     94    760     11400.0  ----- Perl -----  (13 files)
10743    398   1918   8383    125745.0  ----- shell -----  (247 files)
47235   7581  13601  27799    166794.0  ----- Java -----  (203 files)
196449  42227  51455  110347    744447.0  ***** TOTAL *****  (768 files)

Pipeline v2.0

Lines  Blank  Cmnts   NCSL     AESL     
=====  =====  =====  =====  ==========  =======================================
    4      0      1      3        18.0  ----- C++ -----  (1 file)
  119     21      9     89      1335.0  ----- shell -----  (1 file)
23091   3098   3691  16608     99648.0  ----- Java -----  (131 files)
23214   3119   3701  16700    101001.0  ***** TOTAL *****  (133 files)

Pipeline v3.0 sclc (updated nightly)

-- MichaelJPan - 30 Mar 2005

Sources will be released via anonymous cvs once we have determined that a majority of the source code has reached stable status, and once we have established an infrastructure that enables us to closely work users who wish to contribute code patches.

This is anticipated to be complete in September 2005.

Pipeline v.3.0.beta4.0 has been released as of 10 November 2005. The current Java jar package includes both the standalone client and the server.

A copy can be downloaded from the LONI download site.

A list of changes to this version can be found at ChangeLog

Please file bugs and features request at our bug tracking system.

Notes:

• Requires Java 1.4.2 to run (all older versions of Java have reached end-of-life support by Sun)
• This version cannot run on SGI IRIX machines, as SGI has not updated their Java Virtual Machine (JVM) implementation in years, and have only Java 1.4.1.

Generic start

The Pipeline can be started with the following command

java -classpath pipeline.jar edu.ucla.loni.pipeline.Server

Starting on Cerebro

These directions are valid only for internal LONI users

If you have been granted permissions (and a port number) and a port number to run on Cerebro, you need to run the following to start the server

• login to cerebro-sn1
• change your shell to bash
  • type "bash" if your default is not already bash
• source the grid engine settings script
  • type "source /usr/sge/loni/common/settings.sh"
• run the pipeline server on your port number
  • type "java -d64 -jar /cxfs/tmp/pipeline/pipeline.server.jar -port=${port} &" where ${port} is your assigned port number
• exit cerebro-sn1

You can now connect to your server via your Pipeline client. please see ClientDocumentation for more details on this step

P Petrosyan

12.09.08: Implementing pipeline server failover

Z Liu

12.10.08: Pipeline test suite
12.10.08: Pipeline & SGE6.2/DRMAA1.0 integration

K Lozev

12.10.08: Pipeline Web app

J Pierce

• 12.08.08: Installation of cerebro-lsh3.loni.ucla.edu, cerebro-lsh4.loni.ucla.edu

• 12.08.08: Enabled job_resubmit + reschedule_unknown SGE parameter to allow resubmission of jobs on crashed nodes

R Magsipoc

• 12.08.08: Yum update of cerebro-lsh1.loni.ucla.edu, cerebro-lsh2.loni.ucla.edu, cerebro-lsh3.loni.ucla.edu, cerebro-lsh4.loni.ucla.edu

• 12.10.08: Updated /etc/sudoers on lsh1 and lsh3 to allow Pipeline developers to reboot the machine

H Hernandez-Mora

• 12.08.08: Created cerebro-lsh5.loni.ucla.edu, cerebro-lmh5.loni.ucla.edu + compute nodes to test SGE6.2.

-- RicoMagsipoc - 10 Dec 2008

P Petrosyan

Z Liu

K Losev

J Pierce

R Magsipoc

C Hojatkashani

-- RicoMagsipoc - 10 Dec 2008

P Petrosyan

12.09.08: Implementing Pipeline server failover.
12.15.08: Already have the draft for failover without Aliasing. Working on aliases.
12.17.08: Created indexes for Pipeline production server database to make it faster.

Z Liu

12.15.08: Pipeline test suite http://cerebro-rcc.loni.ucla.edu:8102/
12.18.08: Started automated tests on latest SGE 6.2u1 on lsh5 with the test suite.
12.18.08: Configuring pipeline server on mini cluster for applet

K Lozev

12.15.08: Pipeline Web app

J Pierce

• 12.16.08: Ran data cables from pipe-lsh1/lmh1 to core1, reconfigured to 128.97.131.x network.

• 12.16.08: Restricted sshd_config logins to SysAdmins^? and Pipeline Developers on cerebro-lsh1, lsh2, and lsh3.

• 12.17.08: Reconfigured Postfix, NTP, and DNS (with a handful of other minor edits) on pipe-lsh1/lmh1 to reflect the move from the cluster data -> 128.97.131 network. Also, disabled everything relating to RocksReinstall^? (part of last night's issues with accessing the machines).

• 12.19.08: Mounted lmh1:/usr/local to lsh1:/usr/local to resolve an unknown issue with Pipeline file validation, provisioning can be done from either machine in cluster.

R Magsipoc

• 12.15.08: Created /usr/pipelinedb-4.1 on cerebro-rcc and modified cerebro-rcc:/etc/sudoers to allow Pipeline Dev team to sudo as user pipelnvr and start db.

• 12.16.08: Modified /etc/sudoers on cerebro-lsh1 and cerebro-lsh3 to allow Pipeline developers /sbin/reboot, /sbin/ipalias and /sbin/route privs.

• 12.16.08: Created minute-by-minute snapshots for I/O, NFS, processes and memory for cerebro-lsh1, cerebro-lsh2, cerebro-lsh3 and cerebro-lsh4 in /var/log/systat. Configured logrotate to rotate snapshots.

H Hernandez-Mora

• 12.15.08: Configured the developing cluster, cranium-dev. It will use SGE 6.2.

• 12.17.08: Configured the pipeline developing cluster (mini cluster), pipeline-web. It will use SGE 6.2.

-- RicoMagsipoc - 15 Dec 2008

12 august 2005, 10:00am

present: IvoDinov, MichaelJPan, Darby Vanuitert, David Weinstein, Michael Callahan, McKay Davis, J. Davison de St. Germain

Integration of SCIRun into Pipeline

• Why would the Pipeline call SCIRun?
  • Utah - what are some missing features from the Pipeline that SCIRun will provide?
  • UCLA - we are trying to provide family of filters for all task types to enable robustness. An ability to swap in and replace defined "equivalent" filters is a goal we would like to go with the Pipeline
  • Utah - what are some use cases?
  • UCLA - object segmentation, registration, visualization widgets.
• How would the Pipeline call SCIRun?
  • SCIRun can be run as part of a batch process
  • SCIRun can be run as a daemon that accepts jobs
  • UCLA - are there examples of the SCIRun network specification?
  • the network specs are just a subset of the TCL language
    • input files are specified as parameters
  • plan
    • start with using SCIRun as a batch process, build a Pipeline module for it
    • move towards a networking protocol that allows Pipeline to submit packets
    • could also build an Java library that allows Pipeline to directly access the TCL interpreter
    • working towards ability to run SCIRun without interface. implemented and and debugging stage, on the unstable development release.

Integration of Pipeline into SCIRun

• Why would SCIRun call Pipeline?
• How would SCIRun call Pipeline?
  • External applications can call the Pipeline using the defined NetworkingAPI
  • Pipeline can be run as a batch process

Action items

• Install Pipeline
  • problem with the download site
• Install SCIRun
  • Stable (from web) vs unstable (from SVN repository) version?

3.0beta5.4 (released 18 january 2006)

• ability to specify default values for a parameter
• added parameter type "side effect output"
• automatically binds variables to parameters and sets their default value (if any)
• added framework for remote server administration
• added ability to reload the ontology on a remote pipeline server
• added ability to add/edit/remove user accounts on a remote pipeline server
• added ability to add/edit/remove group accounts on a remote pipeline server
• fixed bug so that users cannot open up multiple edit or view windows when it is not appropriate
• there are now 3 sub options on the new menu, on for a workflow, one for a pipeline, and one for a system executable
• when the canvas is right clicked on there are now two new choices, a pipeline, and a system executable
• when editing/viewing a class there are now two separate sections, one for parts and one for subclasses
• when editing arguments and variables that are of type pipeline list, there are now three ways to enter the information, either as strings, using a text box, or importing from a file
• Adding/Editing a variable allows option to create new references and list references from that GUI

3.0.beta5.1 (released 22 november 2005)

• extracted server code to its own package
• corrected bug with "Save as" option
• reset of a monitor over a network now works
• fixed bug with true dataflow parallelization
• fixed bug with non phantom command line parameters
• Added interface for multiple values in variable editing dialog

3.0beta5.3 (released 10 january 2006)

• fixed an OutOfMemoryException
• various GUI fixes

3.0beta5.2 (released 20 december 2005)

• fixed application non loading bug if upgrading from beta5.0
• now validates prior to execution to ensure that resources are bound and resource values are not empty
• execution now correctly progresses on true dataflow
• specification of the server port from command line
• ensured that workflow synchronization upon reconnection works
• fixed message when server connection unavailable
• fixed an issue with loading a v1 pipeline
• Fixed issue with hung interface when attempting to load a non-existant file
• Added a default input field for path in the "Construct URL" dialog
• Changed nomenclature from "Comment" to "Description" in interface
• Fixed issue with extensions not processing events specified as Java Interfaces
• Fixed parameter editing synchronization between parameter class restriction and parameter type
• Disabled ability for user to unbind a resource from an argument
• Fixed slow UI response when opening a contextual menu for an argument bound with multiple values
• Misc. user interfaces bug fixes and enhancements

3.0.beta5.0 (release 10 november 2005)

• Fixed interface delay issues with loading of a pipeline module
• Status of enclosing pipeline is now displayed
• Disabled some "Workflow" menuitems until there is an active module loaded
• Added new "Find" feature in validation window
• Fixed invalid data issues when a value is not assigned to a file variable
• Added better handling of when to allow viewing of argument files
• Fixed issue with status of the nodes not getting updated on abort messages
• Fixed pause and resume
• Added requirement for users to specify output locations
• Workflow synchronization now communicates errors
• Fix to permission verification on Windows clients
• Some fixes to the ontology
• Miscellaneous user interface fixes

3.0.beta4.0 (released 18 October 2005)

• fixes to URL resolution
• SRB integration
• Added ability to view/edit a class definition
• User can now specify an extension for a class definition
• Fixed bug with context-sensitive module double-click
• Combined panels for activation/deactivation of modules and activation/deactivation of parameters in execution wizard
• miscellaneous GUI enhancements and bug fixes

3.0beta6.1 (released 3 february 2006)

• fixed issue with using ${iteration} variable in values of output arguments
• fixed remove administration of users and groups

3.0beta6.2 (released 15 february 2006)

• fixed issue with execution configuration where cache directory was being hardcoded
• fixed ability to specify whether a workflow should overwrite existing files
• implemented framework which adds "help" menu item to all contextual menus

3.0beta6.2 (released 15 february 2006)

• fixed issue with dragging a library function onto the canvas

3.0beta6.0 (released 25 january 2006)

• fixed issue with dragging functions into working document
• XML based account managers now will create default administrative account if none exists
• added multiple file chooser dialog that allows users to see the files they select
• displays error message in server administration accounts panel if remote account managers haven't been loaded yet
• fixed issues with remote user/group management

3.0.beta3.1 (released 13 September 2005)

• various fixes to issues with global graph synchronization
• grid engine integration now possible with commandline tools (eg qsub, any sge version) or drmaa (sge6.0 only)
• close and abort commands across a network now works correctly
• fixes to user connection management gui
• help menu item opens a browser to wiki
• Double-clicking on row in Edit Module panel edits the bound variable
• Enabled deleting modules by pressing delete key
• Editing of values has been simplified in Edit Module panel. Press enter to edit the highlighted value of a parameter
• Fixed creating modules from executable java files
• Fixed problems with saving a module
• Added context dependent double-clicking on the modules
• Added class/connectiontype synchronization in edit function panel
• Allows selecting a directory in the file chooser if the type is Directory
• Fixed issues with allowable connections between parameters
• Edit Module panel is shown after creating a new system executable
• Added ability to add multiple values from a file chooser in New/Edit variable dialog
• Added prompt to kill or cancel when user wants to reset a pipeline

3.0.beta3.2 (released 4 October 2005)

• fixed bug with phantom parameter attribute
• fixed bug associated with occasional packet corruption
• server now supports execution for aliased hostname
• srb integration
• serverside hostname aliasing
• panel for showing a VirtualPipelineMonitor's children and errors
• improved URL and value composition panels in edit variable dialog
• upon creating a pipeline, graph for the pipeline is opened
• fixed bug with opening a recent module
• unified URL construction panels
• removed exported column in edit module dialog
• fixed state of run panel buttons after a pause has occurred
• fixed user/group management bugs
• added a new popup menu item to refresh graph
• added ability to close a workspace from the working list from a right-click popup menu
• added ability to use DOT layout server to automatically position graph modules
• improved layout of user/group management graph using a DOT layout server
• added tooltips for run panel buttons
• miscellaneous GUI enhancements and bug fixes

3.0.beta3.0 (released 31 August 2005)

• Note that this is the first release with a stable server

• Fixed EditFunctionPanel so that changes are not applied until the user presses "OK" (including addition/removal/changes made to parameters)
• Added capability to export a canvas as an image file (jpg or png)
• Fixed canvas zoom issues
• Improved the way a canvas is updated
• Selected cells can be "nudged" using arrow keys. Shift+arrow key results in a larger nudge.
• User/Group management gui bug fixes
• Client now validates user permissions prior to submission and aborts if does not have permissions
• Fixes bug #164 which now gives the user an option to continue running modules on exit
• ability to un/cluster modules
• groups now specify access levels
• unknown class/data types default to "UnknownType" class definition
• Monitor now validates for network connections prior to execution
• various user interface issues

Java virtual machine issues

• Requires Java 1.4.2 to run (all older versions of Java have reached end-of-life support by Sun)
• This version cannot run on SGI IRIX machines, as SGI has not updated their Java Virtual Machine (JVM) implementation in years, and have only Java 1.4.1.
• Some users report having issues with non-official JVMs (eg Blackdown, gcj). If you are having issues running the Pipeline on Linux boxes, please try running the Pipeline using an official JVM from Sun.

Grid engine issues

• On 64bit processors, Pipeline needs to be run with 64bit capabilities enabled. This can be done by issuing the command "java -d64 -jar pipeline.jar"
• Sun's AMD Opteron JVM build 1.5.0_01-b08 is faulty wrt 64bit mode. Upgrade to build 1.5.0_04-b05.

Pipeline module corruption

• For various reasons, a Pipeline module (the .module XML file) may become corrupted and cause issues in the execution. If this is the case, try saving the pipeline as another file using the "save as" option.

User configuration corruption (2005-11-15)

• If double-clicking on the Pipeline.jar file does nothing, and Java is installed, try running the jar file from the command line "java -jar pipeline.jar". If the message you get is "could not instantiate null", there may be a corruption on your configuration. To resolve this, try moving your configuration folder (located at ~/.pipeline/config) to another location and restarting the Pipeline. If this works, please submit a bug and attach a copy of your configuration, so that we can debug the specific issue.

Execution configuration corruption (2006-2-16)

• This is specific to Beta version 6.2 - When you handcode the cache directory, from Application->Options->ExecutionOption, the cache files

may not get created in your new cache directory - Then, please try running beta version (6.3 or higher) and specify a new cache directory, the change is propagated immediately to the application and any subsequent runs of pipeline modules will have their intermediate files stored in the specified directory -

Found a bug?

ReportBugs

---

Got tips for other users? Post them here!

Shape analysis module

This module is developed by Martin Styner at UNC for shape analysis.

Screenshots

Module

• UNC_SHAPE.module:

The LONI Pipeline makes use of JUnit for unit testing.

Existing tests

Current we have the following test classes

* *

Additional tests needed

• loading & saving a pipeline should not result in any diff, unless updating version
• simple align
• side effect output
• virtualized input cardinality
• large lists

Code coverage

• ...

Nightly testing

• ...

Instructions for running tests

> ant JUnitTesting
> java -classpath build/pipeline.junit.jar ${junit.class}

alternatively, to run under the debugger

> ant JUnitTesting
> jdb -classpath build/pipeline.junit.jar ${junit.class}
> run

where ${junit.class} is the your unit test class

Updating modules

There are 2 ways for upgrading from V1. The first is an ad-hoc method; the 2nd is more structured.

In the ad-hoc method, one simply needs to load up any v1 pipeline by selection "Workspace->Load" and choosing the "v1 pipeline" filter, which should show you all the v1 pipelines (those whose filenames end with ".pm"). The pipeline will automatically converted, and the v3 pipeline created (ending in ".module") placed in the same directory, and loaded into the workspace.

In the second method, one can batch convert an entire set of v1 pipelines. To do so,

1. create a text file that lists out the paths of all the pipelines on each line
2. run "java -classpath pipeline.jar edu.ucla.loni.pipeline.test.ModuleSyntaxUpdaterTest ${file}" where ${file} is the file created in step 1
   • this will do 3 things
     1. convert and create new .module files in the same locations as the originals
     2. copy each .module to your .pipeline/modules/ directory, so that they are loaded and ready to be used at startup
     3. create singleton modules that are unknown in the ontology in your ~/.pipeline/modules/singletons directory. Once these modules have been standardized and added to the ontology, running the updater once again will reference them

No permission to read topic UsabilityIssuesApril2006 - perhaps you need to log in?

Groups in the Pipeline application are managed in a tree. At the top of this tree is the root node. All nodes that do not have any parents specified have this root node as the parent.

A group can have multiple parents and multiple children. The root group cannot have parents, and user groups cannot have children, as they are groups created solely for the user with the same login as its name.

To access the User/Group Management dialog, in the main Pipeline menu bar, click on "Server"->"Manage Users/Groups". Users are indicated with a blue node. Groups are indicated with green node. The root group is always a gray node.

An arrow between nodes represents a parent group relationship. The tail end represents the parent group. The head end represents the child user or group. A user or group may have multiple parent groups and must have at least one. The only exception to this rule is the root group, which has none.

• GroupSchedulingPriority^?
• GroupProcessorAllocation^?
• GroupPermissions

• AddUserOrGroup
• EditUserOrGroup
• RemoveUserOrGroup
• SetParentForUserOrGroup

Features

Directory: /nethome/users/{user-name}/{pipeline-dir}/pipeline/pipeline/
Filename: users.xml

Format

<Users>
< User login="your-username" password="your-hashedpassword" group="your-groupname-1;your-groupname-2;your-groupname-n"/>
< User login="jrajendi" password="3rtfpdfunicorn4adobe@loni" group="loni;namic;birn"/>
< User login...   .../>
< User login...   .../>
< User login...   .../>
<Users/>

Variable Definitions

username: : Name of the user
hashedpassword: : The password (hashed) used by the user @ login
group: : The list of groups which are basically groupnames separated by ';' (semicolon)

Rules

1. There needs to be a direct corresspondence between the group names mentioned in this file to those defined in GroupsXml.
   Otherwise, behaviour may not be as expected.

-- JagadeeswaranRajendiran - 22 Mar 2005

UsingHibernate

• PersistentClasses
• Hibernate Mapping
• Hibernate calls
• SourceCodeRequirementsForHibernate
• JUnitHibernateInteraction

SwiXML GUIs

This document provides the instructions for using SwiXML to split the GUI. The end result should be 3 files. These files are

• an execution class
• a GUI XML file, and
• an ActionListener^? class.

Execution

This class holds only methods for non-GUI execution. These methods should be the ones in the original class which do are not GUI related.

XML

This file holds the design of the GUI. Some things that this XML file needs to specify include

• ActionCommand^? attributes in button and menu item nodes

ActionListener

This class listens for events on the GUI, interprets them, and calls the appropriate methods in the execution class. Conversely, this class is responsible for updating the GUI when necessary. For the sake of transparency, this class should maintain the same name as the original (unsplit) class. Some things that class needs to specify include

• import the javax.swing.*, javax.awt.event.*, and org.swixml.SwingEngine packages
• implement the java.awt.event.ActionListener class. This includes the definition of a method actionPerformed (ActionEvent^? ae) which calls methods in the execution class dependent upon ae.getActionCommand()
• hold a reference to the SwingEngine (which we will refer to as swix)
• hold references to all the GUI items defined in the XML file. The XML file simply configures them. This class declares them, and should declare them public
• have a default constructor which should have the following commands
  • swix = new SwingEngine(this);
  • swix.render(xml);
  • swix.getRootComponent().setVisible()
• Furthermore, all GUI elements should register this ActionListener class by calling addActionListener(actionListener)

-- MichaelJPan - 18 Apr 2005

No permission to read topic ValentinoGroupIssues - perhaps you need to log in?

  <owl:Ontology>
    <owl:versionInfo><![CDATA[20050223]]></owl:versionInfo>
  </owl:Ontology>

Currently the version string is "20050412".

A VirtualPipelineCanvas contains a top panel that displays the module monitor information and a bottom panel that displays errors for a virtual pipeline.

-- RonNor - 01 Feb 2006

This page contains screenshots and videos of the Pipeline in action.

All the videos and screenshots have been taken off the site because they were way out of date. We will have fresh content on the new site when it comes up in the next couple weeks.

Topics in Pipeline web:

Changed: (now 05:33)

Changed by:

WebStatistics	21 Nov 2009 - 08:02 - r1.1292	Main.guest
Statistics for Pipeline Web Month: Topic views: Topic saves: File uploads: Most popular topic views: Top contributors for topic save and uploads: Nov 2009 6306 0 ...

SummaryDecWeek2-2008	23 Dec 2008 - 16:35 - r1.13	HugoHernandez
P Petrosyan 12.09.08: Implementing Pipeline server failover. 12.15.08: Already have the draft for failover without Aliasing. Working on aliases. 12.17.08: Created ...

SummaryDecWeek1-2008	16 Dec 2008 - 16:52 - r1.8	PetrosPetrosyan
P Petrosyan 12.09.08: Implementing pipeline server failover Z Liu 12.10.08: Pipeline test suite BR 12.10.08: Pipeline SGE6.2/DRMAA1.0 integration K Lozev 12.10.08 ...

PipelineGridTweaks	15 Dec 2008 - 21:03 - r1.3	RicoMagsipoc
Pipeline Development Group Action Items 2008 December December, 2008, Week 1 December, 2008, Week 2 Back to main Pipeline Wiki

SummaryDecWeek12008	10 Dec 2008 - 03:44 - NEW	RicoMagsipoc
P Petrosyan Z Liu K Losev J Pierce R Magsipoc C Hojatkashani Main.RicoMagsipoc 10 Dec 2008

WebHome	10 Dec 2008 - 03:35 - r1.29	RicoMagsipoc
The LONI Pipeline is a simple graphical environment for constructing complex scientific analyses of data. It provides a visually intuitive interface to data analysis ...

PipelineDownloads	13 Feb 2007 - 23:49 - r1.7	ArashPayan
Pipeline Download the latest release of the Pipeline, and view the release notes for more information.

PipelineFaq	25 Jan 2007 - 19:53 - r1.6	ArashPayan
How can I help to improve Pipeline? The best way to help out is by using Pipeline and giving us feedback. You can sends comments/suggestions to us by email at pipeline ...

VisualGallery	25 Jan 2007 - 19:50 - r1.5	ArashPayan
This page contains screenshots and videos of the Pipeline in action. All the videos and screenshots have been taken off the site because they were way out of date ...

PipelineSupport	25 Jan 2007 - 19:44 - r1.7	ArashPayan
User Links Demos Quick Guide (.pdf) Creating a Basic Pipeline (.mov) Creating a list for Multiple Inputs (.mov) Steps Required for Running a Pipeline on the LONI ...

MeetingMinutesOctober2006PipelineSCIRun	14 Oct 2006 - 04:57 - r1.6	IvoDinov
Pipeline Meetings Pipeline/SCIRun Interaction Meeting (Tele-Conference Call, DATE!) Online Pipeline Calendar CCB Meetings LONIR Meetings Pipeline Meetings Participants ...

MeetingMinutesOctober2006PipelineSCIRun101206	13 Oct 2006 - 01:54 - NEW	IvoDinov
Pipeline Meetings Pipeline/SCIRun Interaction Meeting 10/12/06 Minutes Online Pipeline Calendar CCB Meetings LONIR Meetings Pipeline/SCIRun TCon Meeting (10/12 ...

MeetingMinutes2006October	04 Oct 2006 - 20:13 - NEW	IvoDinov
Pipeline Meetings Pipeline/SCIRun Interaction Meeting (TeleConference CAll, DATE!) Online Pipeline Calendar General Pipeline Meetings LONIR Meetings Pipeline Meetings ...

MeetingMinutes	04 Oct 2006 - 20:13 - r1.19	IvoDinov
Teleconferences TeleconferenceMinutes200508121000 NAMIC teleconferences External meetings MeetingMinutesOctober2006PipelineSCIRun MeetingMinutes200601111300 MeetingMinutes200508161030 ...

DMRequests	13 Apr 2006 - 22:48 - r1.5	ArashPayan
Data Model requests: MINCMATH add –sub –mult –div (could be a file or integers) invert: Calculate 1/x at each voxel (use constant for c/x), can be used with const ...

ResearchPublications	06 Apr 2006 - 03:21 - r1.9	Main.mjpan
The following are publications related to the Pipeline. If you use the Pipeline in your research, please cite one of the following publications. Also available is ...

PanRequests2006	04 Apr 2006 - 19:30 - r1.6	Main.jagadeeswaran
Bugs Mantis #1306 Mantis #1311 Documentation FunctionFactory.java Difference between cardinality and iteration Client Server communication flow (setup, status updates ...

PresentationSlides	30 Mar 2006 - 21:15 - r1.14	Main.mjpan
Past presentations The following is a list of presentations of the Pipeline pipeline.20041210.ppt: This was presented to NAMIC collaborators at the NAMIC Toolkit ...

UnitTesting	30 Mar 2006 - 21:09 - r1.4	Main.mjpan
The LONI Pipeline makes use of JUnit for unit testing. Existing tests Current we have the following test classes Additional tests needed loading saving a pipeline ...

DeveloperDocumentation	18 Mar 2006 - 00:07 - r1.22	Main.jagadeeswaran
DesignGoal SourceCodeRepository CodingStyle NetworkingAPI UnitTesting AddingExtensions (how to create an extension for the Pipeline) UsingSwiXML (how to use XML to ...

ClientDocumentation	18 Mar 2006 - 00:07 - r1.23	Main.jagadeeswaran
InstallingPipelineApplication RunningPipelineApplication SoftwareRequirements PipelineGlossary UpgradingFromV1 LibraryModules PipelineModules ShortUsageTutorial ClientConfiguration ...

ValentinoGroupIssues	16 Mar 2006 - 22:12 - r1.2	Main.mjpan
Daniel Valentino's group's issues Mantis Case #1244 Upon pressing RUN button, an empty window shows up and pipeline won't run. (apayan) Mantis Case #1245 Validation ...

NightlyBuild	16 Mar 2006 - 20:06 - r1.25	Main.mjpan
Nightly builds If you are an internal user, you can grab a copy of the nightly build from /loni/mjpan/pipeline.builds/nightly/ on an internal LONI machine which mounts ...

ReleaseRoadmap	16 Mar 2006 - 20:05 - r1.11	Main.mjpan
Current development schedule: Client/standalone pre-alpha release (Released 30 March 2005) Client/standalone alpha release (Released 18 April 2005) Client/standalone ...

RunPipelineAntScript	16 Mar 2006 - 20:05 - r1.2	Main.mjpan
This script will run the Pipeline using Ant, by including all the external libraries for you in the classpath. You will have to modify some properties ${lib.dir} ...

RunPipelineBashScript	16 Mar 2006 - 20:05 - r1.2	Main.mjpan
This script will run the Pipeline under a bash shell, by including all the external libraries for you in the classpath. Modify $LIB DIR to match the directory where ...

StableBuild	16 Mar 2006 - 20:05 - r1.14	Main.mjpan
Pipeline v.3.0.beta4.0 has been released as of 10 November 2005. The current Java jar package includes both the standalone client and the server. A copy can be downloaded ...

RunningYourOwnServer	15 Mar 2006 - 19:33 - r1.2	Main.jagadeeswaran
To run your own server on cerebro-sn1, using XML-based Authentication, please follow these steps: 1. Make sure your {user.home}/.pipeline/config/ exists on cerebro ...

ServerDocumentation	15 Mar 2006 - 05:07 - r1.10	Main.jagadeeswaran
These are the steps for setting up a server to run. Ensure SoftwareRequirements Define users, groups, and permissions in UserAndGroupManagement ConfigureServerSettings ...

TroubleShooting	14 Mar 2006 - 22:03 - r1.11	Main.mjpan
Java virtual machine issues Requires Java 1.4.2 to run (all older versions of Java have reached end-of-life support by Sun) This version cannot run on SGI IRIX machines ...

ConfigureCacheDirectory	14 Mar 2006 - 19:23 - r1.3	Main.mjpan
Application directory This directory is the location that the application uses. Defaults to the tmp directory of the root file system c:\tmp\pipeline (on Windows ...

JagadeeswaranCodeDocumentation	14 Mar 2006 - 18:59 - NEW	Main.mjpan
Code Documentation (Jagadeeswaran) This section lists the necessary code changes and/or documentation necessary Documentation on adding new configuration options ...

UsabilityIssuesApril2006	13 Mar 2006 - 21:19 - r1.9	ArashPayan
Usability issues These are the usability issues for our demos in April-May 2006 1) Menuing system 2) Edit Module GUI 3) Edit Argument 4) Global Variables window 5 ...

ConfigureServerSettings	10 Mar 2006 - 23:49 - r1.3	Main.mjpan
Pipeline server settings can be configured. ConfigureExtensions ConfigureExecution ConfigureNetworking ConfigureCacheDirectory

DemoIssuesApril2006	10 Mar 2006 - 20:32 - r1.2	Main.jtam
Demo Issues These are the issues that are yet to be resolved for our demos in April-May 2006. Usability We maintain a separate page for usability here. Integration ...

PipelineAbout	10 Mar 2006 - 01:14 - r1.3	ArashPayan
The LONI Pipeline Processing Environment is hosted at UCLA's Laboratory of Neuro Imaging. It has been used quite successfully since its release in the neuroimaging ...

UpgradingFromV1	09 Mar 2006 - 22:34 - r1.5	ArashPayan
Updating modules There are 2 ways for upgrading from V1. The first is an ad-hoc method; the 2nd is more structured. In the ad-hoc method, one simply needs to load ...

WebPreferences	09 Mar 2006 - 22:30 - r1.18	ArashPayan
TWiki.Pipeline Web Preferences The following settings are web preferences of the TWiki.Pipeline web. These preferences overwrite the site-level preferences in TWIKIWEB ...

PipelineModules	09 Mar 2006 - 20:53 - r1.22	ArashPayan
PipelineModules are !XML descriptors of program usage. These descriptors provided information about how a program is run, so that the Pipeline can correctly execute ...

PipelineDevelopers	08 Mar 2006 - 19:57 - r1.5	Main.mjpan
Current contributors Principal investigator Arthur W Toga Technical lead Main.MichaelJPan Programmer Main.JagadeeswaranRajendiran Liaison to NA-MIC Main responsibilities ...

PassingDataInGUI	01 Mar 2006 - 22:01 - NEW	JonMorra
Main.JonMorra 01 Mar 2006 The way that data is passed around in all of the classes that extend !AbstractGUI have now changed Now everything that extends !AbstractGUI ...

CreatingDialogs	01 Mar 2006 - 21:58 - r1.4	JonMorra
Main.JonMorra 09 Jan 2006 PLEASE DO NOT CREATE ANY MORE JDIALOGS IN PIPELINE The !DialogManager is intended to be the only way JDialogs are made. The !DialogManager ...

DownloadSoftware	28 Feb 2006 - 22:01 - r1.15	Main.mjpan
The LONI Pipeline is released under the LONI License. The LONI Pipeline client v.3.0.beta6.3 has been released as of 28 February 2006. A copy can be downloaded from ...

ThreeBetaSixPointThree	28 Feb 2006 - 21:59 - NEW	Main.mjpan
3.0beta6.2 (released 15 february 2006) fixed issue with execution configuration where cache directory was being hardcoded fixed ability to specify whether a workflow ...

ChangeLog	28 Feb 2006 - 21:29 - r1.58	Main.mjpan
ReleaseNumberConvention provides information as to our numbering scheme. Upcoming changes in the next release Current release 3.0 beta 6.3 released 28 february 2006 ...

FunctionDefinitionsIndex	27 Feb 2006 - 23:49 - NEW	Main.mjpan
This is an example of the function definitions index file used by the pipeline to load all the function definitions.

OntologyDocuments	27 Feb 2006 - 23:41 - r1.4	Main.mjpan
Class definitions in the ontology distributed with the Pipeline application ClassDefinitions.owl: FunctionDefinitionsIndex

SourceCodeRepository	27 Feb 2006 - 20:22 - r1.2	Main.mjpan
Source code repository Internal access Make sure you are in the following groups: hackers, piplncmt, piplndev server(s): any system that mounts the /loni file system ...

ChangingConfigurations	22 Feb 2006 - 22:20 - r1.2	Main.jagadeeswaran
Changing Profiles (without using GUI): If the user has multiple profiles, the best way to reflect the changes directly: 1. Each profile has its own directory (with ...

ThreeBetaSixPointTwo	16 Feb 2006 - 02:11 - NEW	Main.mjpan
3.0beta6.2 (released 15 february 2006) fixed issue with dragging a library function onto the canvas

Number of topics: 50

• See 100, 200, 400, 800 most recent changes
• See all changes